Poll after poll predicted the Remain vote would win the day and there would be no Brexit. Almost all of them got it wrong. Thursday’s referendum left the UK preparing to leave the European Union, and many pollsters wondering what went wrong with their efforts to gauge public sentiment.
The polling industry’s crisis of confidence is fueled by several recent failures to forecast major outcomes. From British Conservatives’ victory over Labour in 2015 to Trump’s unexpected dominance of the Republican presidential primary race (you can read celebrity pollster Nate Silver’s 5,379-word mea culpa on that here), well-respected polling outfits seem to keep getting it wrong.
It’s likely an issue of changing technology and communication behaviors. The former “gold standard” polling technique—randomized phone calls to a representative cross-section of the population—is increasingly unreliable, given the spread of mobile phones, government bans on automated calling technology, and declining response rates, reports The Huffington Post’s polling editor, Natalie Jackson. After a statistical review of the three most common polling methods—live telephone, internet, and automated phone—she wrote, ”we can’t really predict which type of polling method will be most reliable in pre-election surveys”
One survey method, however, has proved more accurate than the rest, and is promising to become the future of polling: internet and mobile phone polling. While not perfect, it gives surveyors the insights of personal interviews with the accuracy and efficiency of modern technology. By reaching out through apps, phones, and websites—and collecting a suite of ancillary data—surveyors are finding they can consistently deliver more accurate data.
Online polls, in fact, consistently put the Leave vote ahead by 1.2 percentage points in the Brexit polls, compared to live phone surveys that gave the remain camp a 2.6 point lead, reports The Huffington Post. The same patterns where seen by the Ethics and Public Policy Center, which tracks conservative politics in the United States and Europe, reports The Atlantic.
Betting shops and financial markets, lauded as one of the most accurate prediction methods, consistently put the remain camp ahead and were caught by surprise. Traders woke up on Friday morning to find their confidence in the status quo badly misplaced as global stock markets and the British currency all crashed. One market-derived forecast put the odds of staying in the EU at almost 90%, even after the polls closed, says University of East Anglia researcher Chris Hanretty.
The result was devastating:
“The market is probably more optimistic than it should be,” admits David Rothschild, a researcher at Microsoft and Columbia University who runs PredictWise, a site dedicated to prediction markets. “I have no question the price of staying in the market was inflated, and the price of leaving was deflated” noting that prediction markets put a 25% probability on a Brexit.
The big surprise, he said, came from new polling technology such as the smartphone survey platform PollFish. “The online polls I did went really well,” he said in an interview. “The PollFish polls I did were spot on.”
PollFish, he said, surfaced a nugget of data that almost everyone else missed: The attitudes of undecided votes looked almost identical to Leave voters. Those last-minute deciders proved critical in the Brexit vote. Most pollsters assumed that undecided voters would opt for the status quo (a shaky assumption). Instead, they came down in force on the side of Leave. That was clear in the PollFish data before the voting had finished, says Rothschild.
Those insights are possible, says PollFish, because its surveys can reach an audience of 250 million people through a network of 10,000 smartphone apps. The company’s polls work like an ad network, but instead of ads, they display surveys inside apps users have already downloaded. People can answer a set of questions in return for in-app rewards or a gift card raffle. PollFish charges $1 per finished survey.
“We can reach a few thousand people in a couple of hours. No one else can do that,” says Ray Beharry, marketing director at PollFish, in an interview.
The key to smartphone polls’ insights is the richness of the dataset. Respondents are anonymous, but their data paints an intimate picture of their everyday lives. PollFish can see (and validate) a respondent’s age and gender, and parse its audience by location, interests, websites visited, phone model and year, purchased apps, and more.
There is solid science to support PollFish’s online approach. An 2015 study pulished in the open-access journal PLoS ONE found that the automated and personalized nature of surveys on mobile phones offered better data. “The findings suggest that people interviewed on mobile devices at a time and place that is convenient for them, even when they are multitasking, can give more trustworthy and accurate answers than those in more traditional spoken interviews,” reported the study. It also found text interviews “tell a different story about a population than answers from voice interviews, potentially altering the policy implications from a survey.”
Brexit proved the method’s potential, says Rothschild. “We’ll be approaching polling very differently in the coming years,” he says. “The data from online polling is very exciting. [It] will let us get different data sets and really understand how different demographics are breaking on different questions.”