David Cameron is falling in voters' estimation. Last weekend YouGov found his net satisfaction rating – the number thinking he is doing well, minus the (rather greater) number who think he is doing badly – is the same as Ed Miliband's. No survey has put Miliband ahead on the question of who would make the best prime minister. Nevertheless, the latest figures are causing angst in Tory circles.

Cameron is the Conservatives' biggest asset. In my research on the three leaders last year, only Cameron served as a net attractor of voters to his party. He seemed competent and determined. These characteristics outweighed a faint impression of smugness, arrogance or ruthlessness. Views of Cameron were the main factor in drawing people to the Tories – more important than policies, or negative views about Miliband or Labour. Miliband seemed out of his depth, lucky to have won the leadership against his better-qualified brother. Above all, voters found him "weird". His biggest positive – a hope he would stand up for ordinary people – reflected his party's brand rather than him.

Many people supported the Conservatives because of David Cameron, while many people supported Labour despite Ed Miliband. Deteriorating opinions of Cameron will therefore have a bigger impact on the Conservatives' vote share than worsening views of Miliband would have on Labour's.

The Conservatives do not want to go into an election with the leaders' relative ratings as they are – but it is depressing to hear that plans are afoot to paint Miliband as the Michael Dukakis of British politics: part of a metropolitan elite with no understanding of mainstream concerns. There are at least three reasons why this is a terrible idea.

First, it won't work. Painting Miliband as aloof could backfire on a prime minister who faces exactly the same charge, but that is beside the point. Political insiders cherish the belief that they can present their opponents in a way that deters potential voters, but it simply does not work like that. The Tories do not present Miliband, he presents himself. Voters, in turn, make up their own minds. People did not decide he was "weird" because the Tories told them to.

Second, even if the Tories did make people think less of Miliband, the effect on Labour's support would be minimal. Those who say they will vote Labour are not unaware of his flaws; they just say they'll vote Labour anyway.

Third, an onslaught against Miliband would not improve the Tories' standing, and could damage it further. Voters think parties go on the attack when they have nothing to say for themselves. A view is gaining ground that the government lacks direction, with the number of U-turns – each trivial in itself – suggesting policies are not properly thought through. As Sean Worth, the former Downing Street aide, warned on Tuesday, the Tories risk being associated with cuts but no compensating improvements resulting from public service reform (though willingness to take tough decisions on spending remains one of the party's few positives with uncommitted voters).

Voters appreciate the difficulty of steering Britain through dangerous and uncertain times, and many respect Cameron for getting on with a tough job. There is potential for a Tory recovery if people can see that austerity will pay off, that promises are being kept, and there is a plan for the future.

But the party badly needs a sense of direction. This is what should occupy the Tories, not a quest to make their opponents more unpopular. We already know what the polls look like when a government in a muddle faces an uninspiring opposition: they are the polls we have today.

Lord Ashcroft is a Tory peer and former deputy chairman of the party