When David Cameron stood for the Tory leadership in 2005, it was obvious to anyone who wasn’t a Conservative that he was the best man for the job. Elections are generally won by converting voters from the opposite side and Cameron’s fresh-faced urbanity seemed likelier to achieve that than the Jurassic growls coming from the right of his party.

For similar reasons, Conservatives cheered when Ed Miliband won the Labour leadership. They doubted that his preaching would reach any but the already converted. As it turns out, neither Cameron nor Miliband has made great incursions into enemy terrain since 2010. Fluctuations in opinion polls describe a flight to Labour of Lib Dems who hate coalition with the Tories and a flounce to Ukip of Conservatives who hate Cameron. The swingometer has stalled.

The lack of voter traffic between the big two parties is one reason Labour strategists are less perturbed by Miliband’s dismal personal ratings than most of Westminster thinks they ought to be. It is undeniably a problem that so many people struggle to imagine the Labour leader striding into Downing Street. But Miliband’s aides point out that Cameron’s weakness – the suspicion that he serves rich men first – is especially pronounced among exactly the voters the Tories need to persuade most. The Prime Minister has made no progress in charming people in marginal seats who voted Labour in 2010 and if they don’t switch, the Tories can’t win a majority. Public doubts about Miliband are well-advertised. Cameron’s electoral blockage is better hidden but surgically acute.

In a close election, mobilising loyal support could be critical and here Miliband has an advantage. He isn’t great at poaching Tories but he sure gives Labour people reasons to vote Labour. It isn’t “big tent” politics but, whether he is making an enemy of Rupert Murdoch or the big energy companies, Miliband knows how to make his tribe feel at home in his tepee. Cameron, on the other hand, has confused Conservatives with a mix of provocation and bribery, offending their sensibilities with gay marriage one day, offering them a European referendum the next.

This gets to the heart of the crisis in Tory “modernisation”. It pains those who admired Cameron’s plan for detoxifying his party’s brand to admit that his subsequent flakiness – dismissing environmentalism as “green crap”, for instance – and his lofty manner are themselves toxic. He is such a fluent messenger, it is hard to accept that he may embody the wrong message.

Tory traditionalists have the opposite problem. They are well aware of the gilded complacency that dulls Cameron’s appeal, especially to squeezed households. But they want that flaw to discredit the rest of his project to reform the party. According to this view, the Cameroons led the Tories down a blind alley of cosmopolitan elitism.

The Conservatives are marooned between being the party their leader once wanted them to be and the one they suppose they might have been without him. So Downing Street has settled for unification around a goal all can agree on – trashing Labour. With the help of Lynton Crosby, Cameron’s campaign strategist, the Tories are turning every debate into a warning against the perils of electing Miliband. If there is a problem with the NHS, it is the legacy of Labour negligence. Economic malaise? Labour spent all the money. Immigration? Labour opened the floodgates. The chairman of the Co-operative Bank is alleged to have taken hard drugs? Labour took money from the Co-op so, er, well, never mind the details, there’s something a bit whiffy over there on the left. Hold your nose and turn right.

Vilifying Miliband needn’t stop the Tories nicking those of his ideas they deem workable and popular – hence George Osborne’s recent U-turn on capping pay-day lender charges.

This smash-and-grab approach could work. Labour MPs are easily spooked. Many still see Miliband proceeding into the next election in a rickety red charabanc, good for attracting attention but sure to be run off the road by the Tory armoured Land Rover. Cameron’s approach doesn’t ask people to fall in love with the Conservatives. He just needs them to see voting Labour as a dangerous gamble when the economy is improving.

The weakness in the plan is its sheer nastiness. The vitriol may put some undecided voters off siding with Labour but if there is no positive reason to choose the Conservatives instead, they may just stay at home on polling day. Meanwhile, Tory aggression really fires up Miliband’s core support, which, bolstered by the Lib Dem defectors, might be bigger than the Prime Minister’s.

The Conservatives need millions of new votes to win a majority and it still isn’t clear where they will come from. Cameron once had the potential to deliver that bounty but he squandered it. He may even become an active impediment. His patrician gentility passed for charisma when it was allied to a moderate, optimistic prospectus. Without that, it congeals into arrogance.

The Tories would still be unwise to change leader before the election. (Only a tiny cabal of hardline Eurosceptics even consider this an option.) They could still win or, more likely, stay in power in another coalition with the Lib Dems. The economy is growing. Miliband looks ill-prepared for the total war that is coming his way.

To be back in No 10 after the next election would be victory enough for Cameron. But to the outside observer, to anyone who is interested in Conservatism as a political creed, to anyone who values debate or who recognises that there are principled MPs on the Tory benches, the party looks diminished by its leader. It is being whittled into a crude instrument for the sole purpose of bludgeoning the opposition. You don’t have to be a Tory to believe it should be more than that.