Crisis? What crisis? New analysis shows polling accuracy has been stable over the decades – and might even have improved

It seemed to be a hat trick of polling catastrophes: Brexit, the 2016 US presidential election and the 2017 British general election. But researchers now say that despite popular perceptions, polls are as accurate as they have ever been.

They say a new analysis of political polls shows that errors have not increased over the decades since the 1940s – and might even have diminished.



“A lot of people have claimed that polling is in crisis, that there have been political events that surprised us over the last year … [but] what does the data say?,” said study coauthor Dr Will Jennings of the University of Southampton.

Concerns over polling have not only been aired in the media: a House of Lords committee launched an inquiry into the issue last year.

Lord Lipsey, chair of the committee, said at the time: “We are seeing more opinion polls than ever in recent elections but their greater frequency has not been matched by greater accuracy. In the last seven general elections pollsters have got the result wrong three times.”



The latest research is likely to be a welcome relief to pollsters, suggesting that as far as the big picture goes, polling hasn’t suddenly gone haywire.

Writing in the journal Nature Human Behaviour, Jennings and coauthor Christopher Wlezien, from the University of Texas at Austin, describe how they conducted a series of analyses based on almost 31,000 polls covering 351 national elections in 45 countries between 1942 and 2017.

Looking at 286 elections, chosen because polling began at least 200 days before election day, the team found that errors decreased overall from an average of four percentage points at that stage to under two points by the eve of an election. Errors were larger further away from election day for presidential compared to legislative elections (such as a general election).

“Often in presidential elections voters are still learning about candidates, they are still acquiring information, they don’t necessarily know very much about their policies and characteristics and so forth,” said Jennings. “Whereas in [legislative elections] where people are voting for parties, party loyalties tend to be more stable and durable.”

Looking at polls for the last week of campaigns for 220 national elections in 32 countries over the decades to 2017 – with data for a handful of countries stretching back to 1942 – errors have held steady at about 2%. When the team looked only at the 11 countries that had regular polling over several decades, polling errors were found to have dropped over time.

An analysis of elections between 2015 and 2017 across 11 countries including the UK, France and the US showed that errors in polls for the main candidates or parties were in line with data from the past.



Further analysis revealed that polling errors are larger for major parties or candidates, and for elections based on “winner takes all” approaches rather than proportional representation. However, even when such factors were considered, poll accuracy was found to have remained stable over time, and might even have improved.

The team note the study does not cover referendums such as the Brexit vote, and that all the polls considered were at national level, whereas polling errors for the US presidential election in 2016 are thought to have occurred at the state level. In addition, it is not clear whether all of the elections included could be considered equally free and fair.



Anthony Wells, director of political research at YouGov, welcomed the findings. “It is one of those bits of evidence that we would like more people to be aware of, because we are aware it is not getting worse and it is not a particularly new problem, but of course the public perception is often that it is,” he said. “ I think it is just because obviously people remember the recent errors far more prominently than the errors from long ago. There are famous errors of the postwar US – [the] ‘Dewey defeats Truman’ election, and the polls getting it wrong then, or [in the UK] the 1970 election or the 1992 election.”

Wells said pollsters are often playing catch-up to change their methods of polling to keep pace with changes in society, such as changes in the use of the telephone or internet, and the emphasis on how data is weighted. “Whereas once it would have been incredibly important that samples were representative of social class,” he said, “these days it is incredibly important that samples are representative in terms of education. When pollsters fall behind changes in society you end up with elections where things are wrong.”



