The problem is that poll respondents differ systematically from voters. Pollsters do their best to sample from the population of voters, but their sampling is imperfect. They can ask poll respondents if they are registered or if they intend to vote, but we know that more people intend to vote than actually do. Also, many voters refuse to respond to polls, and suspicion of the mainstream media may increase the rate at which they do so.

Smart poll aggregators like FiveThirtyEight attempt to include these considerations in their interpretation of polling results, but their corrections are incomplete. If Trump voters are more likely to hang up on pollsters, then how should a forecast impute the preferences of non-respondents? Historical data may provide a partial guide, but if Trump supporters’ suspicion of the media has increased over the last four years, then historical guidance will be imperfect.

Given these problems, should we abandon our reliance on election polls? No. We believe that it is neither practical nor desirable to give up on election polls. Asking likely voters about their preferences will continue to be one of the best ways to gain insight into the interests and intentions of the voting public. For all their faults, voter surveys help politicians understand which policies and issues the voters care about and they help political campaigns target their efforts. Instead, we ought to get smarter about how we interpret poll results.

Our goal is for those reading about polls to understand their limitations beyond the errors that are traditionally reported, and to adjust their interpretations accordingly. How much would you have to expand a poll’s margin of error so that it included the election result 95% of the time? Our data suggest that, for polls taken the week prior to the election, margins of error would roughly have to double in order to include the actual election result 95% of the time. For earlier polls, the margins of error would have to expand even more.

Poll results are useful, but they should be interpreted with considerable skepticism. They are overconfident for many of the same reasons that humans are. Both are frequently wrong because they do not know what they do not know. Our knowledge of the facts is biased in ways we fail to completely appreciate and correct. We believe too fervently in our own views, including our political views, and are too sure that we are right. All of us, not just pollsters, would do well to expand our margins of error and accept the possibility that we might be mistaken.

📬 Sign up for the Daily Brief

Our free, fast, and fun briefing on the global economy, delivered every weekday morning.