Hidden away in Ed Miliband’s unorthodox speech about the perceived failings of his leadership is a more defiant claim: he asserts confidently the battle of ideas is what matters most in politics.

On the surface the claim is odd, and reinforces the caricature of Miliband as a geek. Most voters do not pay attention to politics, let alone ideas. Few will take Thomas Piketty’s book on Inequality or Michael Sandel’s thoughts on failing markets with them on holiday.

But in politics the battle of ideas is not a contest between intellectuals or geeks – the equivalent of Monty Python’s football match, where Marx comes on as a substitute and Socrates scores a late goal. It is more a case of the electorate’s mood moving ahead of most mainstream politicians, who tend to look back to the immediate past for guidance. Who has the ideas that best adapt to a changing tide?

One of the mysteries in the current political situation is why the Conservatives are not well ahead in the polls. The economy is growing. The IMF pays homage. Miliband endures a media onslaught that would normally doom a party as well as its leader. David Cameron is a confident performer, and his persona is mediated through mainly supportive newspapers. If this were the pre-election summer of 1982 or 1986, we would look at these political ingredients and predict, accurately, a Conservative landslide in the year to come. Yet not even the most ardent of Miliband’s critics predict such an outcome.

There is a single explanation and it relates, as Miliband suggests, to ideas. For a party to win this contest, its ideas must match the mood of the times. In addition, the policies that arise from those ideas must form part of a coherent political project. Speaking in the Commons in 1991, when the Conservatives were miles behind in the polls, Nigel Lawson, the former chancellor, noted confidently: “The Conservatives are still winning the battle of ideas and therefore will win the next election”. They were, and they did.

The 2008 crash marked a turning point as big in its way as the “sea change” that Jim Callaghan detected in 1979, an unstoppable force that threw him out of office. In his speech a week ago Tony Blair stressed that what had not changed after the crash was as important as what had – a crucial, if obvious, point. But even Blair admits that much tougher regulation is one of the necessary responses, which is a very big challenge in itself. And this guides us to what has changed since 1992, when the Conservative party last won hearts and minds.

After 2008 there were in effect two recessions. Now unemployment is falling but insecurity remains; the jobs are part-time or freelance; and the young cannot afford to buy homes in the booming areas, where most of the jobs are on offer. England is not about to become Sweden, but atomised voters seek binding agencies, clearly accountable, in the era of mistrust. The signs are all around us, from polls suggesting big support for state involvement in the railways and failing markets to hostility across the political spectrum towards some foreign takeovers of British companies.

Cameron is a smart reader of politics, and he almost got the need for change before the crash. But his party and his ideological upbringing – and the fact that he secured the leadership with little experience – makes him ill-equipped to win the new battle of ideas. As a result he flicks the same switches but finds there are not the same political dividends. Instead there is trouble.

The demotion of Michael Gove is explored purely in terms of personality and soap opera. But Gove is charming personally and a brilliant interviewee: discursive, curious and tonally varied. The reason Gove was unpopular is that the policies alarmed not only the bad teachers but also the good ones, and parents too. Those I speak to express fear and frustration about the fractured chaos on the ground rather than about Gove’s personality.

Similarly there was nothing wrong with the ministerial personality of Andrew Lansley. But his big idea to fragment and fracture the NHS terrified patients as much as providers. Again, George Osborne is by far the most fully formed politician in the cabinet. But some of his 1980s-style economic policies have led to an economic recovery that does not extend to seats in the north-west and north-east that Margaret Thatcher won with her eyes closed. Thatcher won in 1979 because she was applying new ideas as the tired Labour government pulled old levers to find that they did not work any more. Now it is her levers that are old, and the current government pulls them in a different era without the same success.

At his peak Tony Benn used to argue that it was the policies that mattered more than the personalities. He was right, but did not realise that he had an engaging personality and policies that were wholly at odds with the electoral mood at the time. The current government is full of decent, and in some cases engaging, characters, but it is failing to win the battle of ideas. That is why, in spite of economic growth and a media out to get Miliband, the next election will be close.

Twitter: @steverichards14