A systematic bias in the way people were selected to take part in opinion polls before the general election is emerging as the most likely reason why the industry failed to predict an overall majority for David Cameron in May’s general election.

Analysis undertaken by polling companies, including YouGov and ICM, of what went wrong in May has found that that a relative over-representation of politically engaged young voters produced a forecast that flattered Ed Miliband. Conversely, the over-70s – who broke heavily for the Tories – were under-represented in YouGov’s internet panels.

The findings come before the publication in January of the initial findings of an independent study for the polling industry, led by Prof Patrick Sturgis of Southampton University, to examine why so many failed to predict a majority win for the Conservative party in May.

Polling puzzle: how response times may explain election surprise Read more

YouGov research into its election errors – it underestimated Tory support by 3.7 points and overestimated Labour by 2.8 points – identified an excess of politically engaged young respondents. Because of their age, they were disproportionately Labour but – because of their interest in politics – they were also more likely than the rest of their age group to turn out and vote.

ICM has long afforded a weight to each age bracket (18-24, 25-34 and so on) to match the group’s size in the census but is now concerned that the individual respondents in each bracket may not be representative of that group.

The polling company is reviewing its weighting scheme so that sample groups reflect turnout likelihoods among young and older people, and between people at all points on the social spectrum. This, together with somewhat stronger assumptions about the behaviour of respondents who decline to say how they will vote, has the average effect of reducing the Labour vote share by about two percentage points and increasing the Conservative vote share to a similar extent.

Another polling firm, ComRes, is focusing its election postmortem on turnout, which it found was lower at the ballots in May (65%) than was implied in its poll. ComRes already had an adjustment for respondents’ tendency to exaggerate their likelihood to vote, but it has now introduced a more refined turnout model that allows it to adjust differentially by age and class.

Sturgis’s study, in which he led a team of eight expert colleagues, principally fellow academics, looked at several scenarios to understand what went wrong. A late pro-Tory swing; error in the final technical adjustments that pollsters apply, through weighting and filtering schemes; and particular categories of missing voters, who no longer talk to the pollsters at all, were all looked at. There were also awkward questions for the pollsters, including whether the “bunching” of final surveys reflected a lack of nerve on each firm’s part to be out of line with the others.

Sturgis had previously indicated to the Guardian that online polling, which draws samples from large panels of volunteers, might be a flawed method, saying: “Not everyone puts themselves forward to take part, and those who do will not be representative.

“Many households are increasingly reluctant to pick up their landline, and a growing proportion of young people in particular rely exclusively on mobile phones.”

A firmer conclusion of systematic bias in the way people are polled has developed, particularly after the late publication of the one poll to have called the 2015 election right – the face-to-face British Election Study (BES) survey.

Instead of relying on the sorts of people who can be persuaded to pick up a landline telephone or put themselves forward for an internet panel, the BES survey knocked on doors selected by a genuinely random sampling. It accurately put the Tory advantage at seven percentage points. Although this was a post-election “recontact poll”, pollsters such as ICM and YouGov tried recontact on their own final samples and found their results did not differ – they still failed to put the Conservatives sufficiently far ahead.

Comparing the detail of the BES survey with the flawed internet and telephone polls pinpoints more specific shortcomings where young people are concerned. The pollsters talked to too few young people and then responded to this deficiency by increasing the weight of those youngsters that they did survey.

The signs of common themes emerging from the various pollsters’ tweaks, as well as from the BES, is good news for the industry, as well as for all those who see value in having access to imperfect information – rather than no information at all – about what their fellow citizens think.

YouGov has outlined various steps that it will take to address these issues, including increasing the number of apolitical people on their panel and boosting the sample targets for older age groups.

Although the election destroyed a great deal of confidence, the pollsters can point to a few “tests” passed in the months since. YouGov correctly anticipated the scale of Jeremy Corbyn’s victory in the Labour leadership contest, while all the main pollsters in France accurately predicted the recent regional vote. Meanwhile, this month’s Oldham byelection showed what an election without polls look like. Pre-election expectations of a close vote were set by reportage, cosy briefings, betting markets and gut instinct. They were all wrong.