It was not a good night for Israel’s pollsters, but with so many factors in play the job of forecasting is both complicated and formidable

It wasn’t a good night for Israel’s pollsters. The average of pre-election polls showed Binyamin Netanyahu’s Likud party on 21 seats, trailing the centre-left Zionist Union led by Isaac Herzog by four seats.

Exit polls didn’t do much better. Once voting ended, these had the two blocs tied on 27 seats each.

With all the ballots nearly counted Likud is set to win 30 seats, and Zionist Union 24 – all but securing Netanyahu a fourth term as prime minister.

Considering that Israel’s parliament is only 120 seats, polling was well wide of the mark.

Why were the polls so wrong?

Hayes Brown (@HayesBrown) My #analysis of Israel's election results pic.twitter.com/Jm8h749ssl

It’s complicated, but let’s start with the polls.

First, it is worth keeping in mind that polls aren’t predictions, they’re snapshots – they tell us what people are saying right now. In Israel polls cannot be published in the final four days of the campaign, and it is possible voters changed their minds in those final 96 hours.

Israeli polls have in the past missed surges in party support over the final days of a campaign, and they regularly discount undecided voters. A poll conducted by Haaretz in late February found 21% of voters totally undecided, and more than one third of supporters of Zionist Union and Likud saying they could change their mind.

Further evidence of this late switch might be one reason that when looking beyond the Likud result, the main difference between the polls and the election result is a shift in votes from the parties on the right – Jewish Home and Yachad (which even failed to enter parliament) – to Netanyahu. The PM’s lurch to the right might have worked. In fact, looking at the polling and results of the other parties, the differences are not so dramatic.

Second, polls carry a margin of error and levels of confidence. For example a poll of 1,000 people has a margin of error of about plus or minus three points and a confidence interval of 95%. This means that 95 times out of 100 the figure in a poll will be within three percentage points of what it would be if you surveyed the entire population.

The Israeli polls were clearly well outside the margin of error. However, a critical point here is that technically the margin of error is based on a random sample (of 1,000 in the case of our example) and does not account for significant factors like how the sample of the population is designed (eg how representative it is) and the degree and methods of weighting applied to the raw figures.

The design of a robust sample is very much tied to the quality of demographic data available (eg an up-to-date census).

Third, there are of course other factors which can influence polling figures, such as voters not wanting to share their political views, or shy voters (who would rather not say they are voting for a specific party). Most pollsters will account for these behaviours – if they are a common trend – based on past polling performance and electoral outcomes; nevertheless, these add to the uncertainty.

And there is of course also a element of possible (statistical) randomness and no recognisable patterns in behaviour.

It is virtually impossible to say whether the difference between the polling and the election outcome is down to a change in voters’ intentions or in a systematic error in the polls themselves.

Facebook Twitter Pinterest In 1992 UK polls forecast Labour’s Neil Kinnock to become prime minister. The Conservatives and John Major won the election by more than seven percentage points. Photograph: Nick Skinner/Associated News/REX

The fact the exit polls too were out of sync – generally speaking, in the same ways – with the result doesn’t help either, and possibly points to a mix of reasons.

Exit polls are extremely complicated (and expensive). The main difference between a regular poll and an exit poll is that the former asks “who will you vote for?” while the latter asks “who did you vote for?”.

Exit polls are based on thousands of interviews carried out outside polling stations. Anything else is not really an exit poll.

The starting point to consider when conducting an exit poll is the choice of places to poll, as only a 100 or so people are interviewed in each. The challenge is selecting constituencies and polling stations that together are somewhat representative of the country as a whole. This is difficult because, alongside demographic factors, results vary between elections (a district representative of the result at the last election may not be at the next one).

As with regular polling, designing the sample population, and the quality of the data available when doing so, is key.

For example in the UK, Ipsos Mori polls more than 10,000 people in over 100 constituencies.

The second task is to model the collected data into a projection by taking into account various factors such as demographics (which exit polls helpfully collect alongside voting information), past voting and exit poll patterns, differential swing and turnout between different locations, and the likely outcome all this pooled information from the polled constituencies implies for constituencies elsewhere.

It is important to note the key word here is “likely”: an exit poll provides an estimate result. It assigns a probability to outcomes across the country based on the data collected.

The Mori exit poll was spot-on in both 2010 and 2005, however a BBC exit poll in 1992 forecast a hung parliament and in the event the Conservative party secured a majority.

With so many factors in play, the job of forecasting is a formidable one. However, because past behaviour so strongly influences present day modelling, errors can provide key lessons critical to improving how we predict elections.

