David Cameron’s victory was clear-cut, says Andrew Cooper , but now he should prove he does not need the Lib Dems for compassion and progressive politics

A frustrating consequence of the polls turning out to be wrong is that no one really knows exactly what happened, in terms of the churn of votes and the shifting between parties, in the years and months leading up to the general election.

At the 2010 election, the polls were fairly accurate – well within their immutable margin of error – while slightly understating Labour support and overstating the Liberal Democrats. We can now conclude that over the past five years the polls started (gradually, presumably) to present a much less accurate picture of the strength of the parties.

At last year’s European parliament election, the polls all put the Conservatives too low and Labour too high. With the wisdom of hindsight, that was a warning signal. The variance was again at about 2%, within the statistical margin of error. But the polls weren’t scattering, as they should, either side of the actual level of party support. They were all erring in the same direction. Each pollster’s findings looked acceptable in isolation, but the total picture should have told us something wasn’t right.

Inside the campaigns: cavalier Lib Dems were hurt by dowdy Roundhead role Read more

The story the polls tell of the last Parliament is that, after a brief post-election honeymoon boost, the Conservatives were on about 36%, a shade below their 2010 election result, from the beginning of 2011 until the “omnishambles” period, in the spring of 2012. From that point right through to last week’s election, the polls scattered either side of a Tory vote share of 32%–33%. For the past 38 months, according to the polls, Conservative support has been within 1% of 33%. In the period since the 2012 budget, over 1,000 voting polls were published: all but one of them put Conservative support lower than the 38% they got across Britain in the 2015 general election; the vast majority put them 4%-6% lower.

Labour’s tale of the past five years, according to the polls, is a rise in support to just over 40%, taking them into a clear lead over the Tories, at the beginning of 2011 – and holding at that level until early 2013. Since then, throughout the second half of the parliament, the polls showed Labour support steadily slipping away – slowly but persistently.

By last autumn, the double-digit lead that the voting polls said Labour had held for a year in the middle of the parliament had dwindled to nothing. Since late last year, the polls have shown Labour and Conservatives deadlocked, with less than a percentage point between them.

It may be tempting … to gorge the true-blue wish​list … That would be a foolish misreading of the mood of the country.

There was no sudden late clustering, as some have asserted. For months leading up to election day, the polls were scattering around the central prediction of a virtual tie, with the ensuing certainty of a hung parliament and political chaos. Then came election night: a clear-cut win for David Cameron’s Conservatives, with a 6.5% margin over Labour. We must assume that the polls weren’t just wrong at the end, but had been wrong for a long time.

We have been here before. The polls were at least as wrong in 1992. An inquiry will be held and changes in methodology will no doubt follow, as in the wake of 1992.

There is speculation about very late swing and voters deciding to vote Tory only at the moment they put pencil to ballot paper. There is some evidence – empirical as well as anecdotal – to support that theory. Many also believe that 2015 saw the return of the “shy Tories”: people who know they’re going to vote Conservative, but can’t bring themselves to say so.

If this turns out to be the case, the pollsters will have to work out what to do about it – and the Conservatives will have to consider the even more challenging question of why voters should be so reluctant to confess that they’re going to back them. It implies that there is a breach between the values of voters and those they perceive in the Conservatives. It suggests that while they can win – just – when voters conclude they are the least bad option, the underpinnings of this election victory are not that strong.

Inside the campaigns: cavalier Lib Dems were hurt by dowdy Roundhead role Read more

A lot of voters were not very enthusiastic about concluding that they would vote for the Tories. With a more competent and Blairite Labour leader, many of these voters may have resisted the conclusion altogether.

It may be tempting for the Conservative government to gorge for a while on the true-blue wishlist the Liberal Democrats prevented them from pursuing. That would be a foolish misreading of the mood of the country and an overestimation of the underlying strength of the Conservative position.

This is David Cameron’s chance to prove that he doesn’t need Liberal Democrats to initiate progressive policies, to demonstrate that he is, as he always said, a modern, compassionate Conservative. The Conservative party needs to follow his lead. Doing so would surely help ensure that if there were shy Tories in 2015, there won’t be in 2020.

Andrew Cooper is a Conservative peer, Lord Cooper of Windrush, a director of the polling company Populus , and a former strategy director at No 10