At 10pm on 7 May last year, Martin Boon, the head of the polling company ICM, spoke for his entire industry in a two word tweet: “Oh, shit.”

Our most recent general election was the most polled in British history. As we found out when the returns rolled in, quantity was not a guarantee of accuracy. All 11 of the polling companies had told the country that Labour and the Tories were neck and neck. What we got was a Conservative vote share a substantial 6.5 points ahead of Labour and a surprised David Cameron smiling at the door of Number 10.

Did the failure of the pollsters actually matter? Copious egg was smeared on their faces with collateral splattering on the chops of journalists – myself included – who invested too much faith in them. But the only thing hurt by that embarrassment is our pride. The important question is whether the inaccuracy of the polls had a material effect on the outcome of the election. I know a lot of Labour and Lib Dem people who think that it did. They say that polls did not just call it wrong. By doing so, they changed the result.

The polls drove incessant media conjecture about what might happen in a “messy” hung parliament and that played to the advantage of the Conservatives. It gave the Tories the context to launch a propaganda barrage designed to stir the fears of Middle England that a Miliband minority government would be the puppet of the Scottish Nationalists.

Had the pollsters detected that the Tories were more than six points ahead, David Cameron would not have been able to run that campaign. Had we known that the Conservatives were in the lead, much more attention might have been paid to what a majority Tory government would get up to and that might have placed the Conservatives under more intense scrutiny.

Lib Dems are particularly convinced that the spectre of a Miliband-Sturgeon axis was responsible for costing them a lot of their seats as scared voters ran into the arms of the Tories. It might even be paradoxically true that by forecasting a hung parliament, the polls helped to produce a Tory majority government. I think there is something in this, but the trouble with the hypothesis is that it is just a hypothesis. Since we can’t rerun the election with accurate polling, it can’t be proved.

That hasn’t stopped some voices from responding to the polling failure by demanding a ban on their publication in the days before an election. That is a rotten idea. It would be anti-democratic, unfair and it wouldn’t work anyway. In a free society, it should not be illegal to collect opinions and publish the results. Another objection to a ban is that it would be partial. A privileged minority, commercial interests and the political parties themselves would still conduct and have access to private polls. In any case, a ban looks highly impractical because it could not prevent websites abroad from publishing polls.

The second reaction to the polling flop has been to say that we should all take a collective vow to ignore them in the future. We should clearly be warier of treating them as though they are unimpeachable oracles, but dismissing them altogether would be silly. It is useful to know what voters are thinking about the parties, their leaders and their policies, especially if we want our politicians to be responsive to the public.

And if we want to know what people are thinking, opinion polling, for all its problems, is less flawed than any other method of trying to read the country’s mind.

If not polling, what? Conversations with random folk down the pub? Tea leaves? Tarot cards? Chicken entrails? Recruiting Mr Spock to mind meld with the nation? What your mates are saying on Facebook and Twitter? I really wouldn’t rely on that as an accurate guide to what the country as a whole is thinking. Depending on the politicians to get it right would be another reckless way of trying to forecast electoral futures. Ed Miliband thought he was going to win last May; David Cameron believed he was out; most Lib Dem MPs reckoned they were going to hold their seats. Wrong, wrong, all wrong.

So opinion polls still matter, which means that it also matters that an effort is made to make them more reliable. To be fair to the much-abused pollsters, they have been conducting a serious endeavour to find out what went awry. The post-mortem by the British Polling Council will be published this week. We have a good idea of what it will say because several of its member companies have already offered their explanations. They vary a bit depending on which one you talk to, but there is a rough consensus about what went wrong. It boils down to this. The pollsters miscalled it because their samples weren’t an accurate representation of who would turn up on polling day and how they would cast their ballots.

Put at its most simple, the pollsters had too few Tory supporters in their quotas and too many Labour supporters.

Stephan Shakespeare of YouGov says his company failed to reach enough over-70s and because this demographic leant heavily to the Tories it led to an underestimation of Conservative support. This sounds plausible. More elderly voters are less likely to be connected to the internet, which makes them harder to reach by companies that do their polling online. Older voters also tend to be more resistant to responding to phone surveys, which is a difficulty for the companies that conduct their polling that way.

YouGov also thinks it had too many of a particular kind of younger voter in its samples, which is a problem also identified by Ben Page of Ipsos Mori. The number of the young who would come out to vote Labour was overestimated because the pollsters’ samples included too many of the young who had high levels of interest in politics, a very Labour-leaning group, and too few of the less engaged, a group much less likely to vote at all.

This was compounded by a well-known challenge to polling: the voters themselves. Some of them have a terrible habit of declaring that they are definitely going to vote only to fail to turn up on the day. At the four previous general elections, the differential was about 10%. So pollsters adjusted for that. They were caught out because the differential leapt to 16% in 2015, says Ben Page.

Is there a fix? Professor John Curtice, the president of the British Polling Council, argues that the solution is to switch to random probability samples because these are better at getting to the “hard to reach” voters. He cites the method used by the British Social Attitudes survey when it asked its respondents to say how they had voted. The result wasn’t perfect – it wasn’t right about the party shares – but it did come up with a Tory lead of six points, close to the real outcome.

The explanation is that the random selection of voters doesn’t take a refusal for an answer when it encounters the more elusive segments of the electorate. The BSA survey makes repeated attempts, in some cases as many as nine, to get the people selected for interview to participate. That is also the problem with this method. It requires a lot of time and a lot of money. It would not be practical during an election campaign when you want to know how opinion could be shifting from day to day.

I suspect the pollsters’ job is getting harder. The contagion of cold calls from the PPI pests and accident claim chancers is surely increasing the resistance among voters to answering their phones to strangers. Some political strategists also have a hunch that the public have grown quite savvy about polls and some now use them to “game” politics.

A voter will say one thing to a pollster in order to send a message while planning to do another when they actually cast a vote. That could be an issue in the polling in the run-up to the EU in/out referendum. If you are nervous of leaving it, a category into which a big chunk of the electorate would fit, you might well tell a pollster that you would vote to leave before casting a vote to remain once in the polling booth.

My fix is to be better at remembering my own rule in the future. I have often argued here that the headline numbers of the polls are not as important as the key details, especially the parties’ ratings for leadership and economic competence. Had I paid more heed to my own advice, I ought to have seen that David Cameron would be back in Downing Street. In the run-up to the last election, all the pollsters reported that Labour was being trounced by the Tories when voters were asked which party they most trusted with the economy and which had the best candidate for prime minister. No party has ever won when it has been behind on both these critical questions.

If we had concentrated on those numbers, the pollsters were telling us who was going to win the election.