On 12 December the country faces its fourth election in a little under five years and, with the stakes so high, the opinion polls are the subject of a great deal of scrutiny and speculation. It is fair to say that pollsters have not covered themselves in glory recently, with failures of varying magnitudes in the last three national elections. So can we trust the polls in 2019, and what are some of the issues that poll watchers should look out for?

In broad terms, the answer to the question of whether the polls can be trusted is a qualified yes. Will Jennings, professor of political science and public policy at the University of Southampton, has constructed a database of polls from the final week of every UK election since 1945. This shows that, on average, the polls have come within around two percentage points of the actual vote share for each main party. Given the technical difficulty of accurately estimating vote shares, that’s a pretty impressive record.

However, two important caveats are in order. First, this is a long-run average and in some elections, such as 1992 and 2015, the error was larger. Second, the average error on the difference between the two main parties is larger at four points, and this is the more important number for predicting which party will form the government.

Even if the polls get the national vote share correct, there is no guarantee they will be a good guide to which party (or parties) form the next government

Underneath these long-run averages, there’s also the confusing phenomenon of different polls giving very divergent readings. Last week, for example, we had YouGov giving the Tories a 14-point lead over Labour while ICM estimated a lead of just seven points, even though both polls were conducted on the same day. How can this be? Well, partly it is just the random variability that we should expect to see across different samples. Indeed, random sampling variability is the very reason that we should not put too much interpretive weight on any single poll as it could be an outlier or statistical oddity.

But it is also to do with the different ways pollsters collect their samples and the decisions they have to make about factors such as predicted levels of turnout, and how to treat undecided voters. Ideally, there would be some way of figuring out now which are the good and the bad polls. But that would require some other more accurate measure of the future election result; and if we had that, we wouldn’t need polls.

The primary methodological challenge for pollsters is to produce samples that are representative of the population of voters, without knowing exactly what that population looks like. The failure to produce representative samples was what led to the 2015 polling disaster, when the polls all predicted a hung parliament but the Conservatives won a clear majority. The basic problem then was that the polls contained too many Labour voters and not enough Conservatives.

David Cameron and his wife Samantha return to No 10 on 8 May 2015 following the Conservatives’ election victory, which the polls failed to predict. Photograph: Reuters

Pollsters try to make their samples representative by applying statistical weights, adjusting the sample so that it looks like the voting population in terms of age, sex, education and so on. Most polls are also weighted to the vote share at the last election, which makes sense if the primary objective is to make the sample as politically representative as possible. However, past-vote weighting can be problematic in situations where many voters have switched parties and new parties have emerged since the previous election. This is, of course, exactly the situation we find ourselves in now.

And, even if the polls get the national vote share correct, there is no guarantee they will be a good guide to which party (or parties) form the next government. The high rate of tactical voting, switching between parties and uneven geographical spread of party support across the country mean that translating national vote shares into seats using uniform national swing – which assumes the direction and magnitude of change in party support at the national level will be consistent across all constituencies – is likely to produce inaccurate results.

The difficulty of translating national vote shares into seats is one of the reasons that poll commissioners are increasingly turning to MRP (multilevel regression and post-stratification). If you haven’t encountered MRP yet, you probably will over the course of the coming weeks. MRP is a comparatively new methodology that predicts the party choice of individual voters using a statistical model for a very large national poll sample. It then applies local-level weighting to the model predictions to produce estimates of the result in each constituency. The advantages of MRP are likely to be particularly well suited to this potentially five-horse race.

MRP was used by political scientists Ben Lauderdale and Jack Blumenau ahead of the 2017 general election to accurately predict a hung parliament. Yet, as with any statistical model, MRP is prone to a range of errors, and there seems to be an unrealistically high level of confidence in it.

A final thing to look out for is the exit poll, the result of which is announced with great fanfare and drama at 10pm on the day of the election. A new methodology, introduced in 2005, has been so accurate that it has come to be regarded as tantamount to the result itself. It works by comparing changes in party support since the last election across a carefully selected sample of polling stations, then extrapolating these changes across all constituencies. But despite its excellent record, the exit poll is based on a sample of responses to a question rather than actual votes cast, and is therefore subject to many of the same challenges as conventional polls. Of particular concern this time will be the likelihood of a higher rate of postal voting, which isn’t counted in exit polls, and the emergence of the Brexit party since the last election, for which pollsters have no baseline. Could 2019 be the first time the new exit poll methodology delivers the wrong result?

The 2019 election promises to be one of the most complex and difficult to predict in living memory, so this Christmas, please spare a thought for the poor pollsters.

• Patrick Sturgis is professor of quantitative social science at the London School of Economics. He chaired the British Polling Council inquiry into the failure of the 2015 election polls. The LSE and the British Polling Council are hosting an event on polling on 27 November