In 2005 when David Cameron was elected leader, the Conservatives had been on the ropes for more than a decade. His modernisation programme did enough to make him prime minister, but only in coalition. The Tories gained the most seats since 1931, but started from a place where too many voters thought the party was "not for them".

It's easy to forget how much Cameron has changed his party. Would previous leaders have backed gay marriage, introduced a bank levy, increased capital gains tax, carved low-wage workers out of the public-sector pay freeze, or introduced a pupil premium for poorer children?

Yet today the Tories urgently need a new round of renewal. While progress has been made, Tory modernisation has not healed the party's worst wounds. It is still seen as the party of the rich. It does badly in urban areas, particularly outside the south-east. And that's partly because it does so badly among Britain's biggest ethnic minorities.

Obviously Cameron should ignore calls from the usual suspects to lurch rightward. But the solution isn't as simple as tacking to the centre – after all, most voters have tough views on crime, welfare and immigration. What's needed is a blue-collar modernisation, focused on the north and Midlands.

Voters are a lot more cynical than they were in 1997. Inheriting the biggest deficit since the second world war means that Tory modernisers can't easily demonstrate change by promising big increases in public spending. While Labour's lead is smaller than you might expect given the mega deficit, the Tories need to be about 9% ahead of Labour in the polls to win a majority. They have been averaging about 10% behind. So with no change of direction there's no chance of a majority.

In 2010 Cameron managed to get more ethnic minority MPs elected. There is no magic policy bullet that will get the party more ethnic minority votes – non-white voters aren't a homogenous group, and don't want to be pigeonholed. But polling for Lord Ashcroft suggests some minority voters are put off by the party's historic baggage. As a small start the prime minister should explicitly renounce Norman Tebbit's "cricket test". Supporting Pakistan or the Windies at cricket is no more evidence that someone has failed to integrate than wearing a kilt to a wedding is proof of Jacobite sympathies. You can be intensely British, but want to hold on to your family history.

Polls show the answer to the "party of the rich" label is not to whack the rich but to focus ruthlessly on driving down unemployment, helping people earn more, and cutting the cost of living.

Further welfare reform is an important part of the answer. But so is more help. Because chronic worklessness in Britain first became established during the Thatcher years, the Tories have a special responsibility to fix Britain's unemployment blackspots. Cameron should launch a war on unemployment. That could involve increasing working tax credits, to better reward those who work hard on low incomes. It could mean cutting employers' national insurance contributions for people on low wages. In the most run-down towns it could involve breaking with free-market orthodoxy and backing government job creation.

How to pay for all this? The most popular source of funds would be to cancel the planned increase in foreign aid spending. The prime minister won't do that, of course; and while I don't agree, I respect his reasons for sticking with it. What about middle class benefits? Do we really need to give child benefit to households that are better off than average? We could redirect funds from other benefits that discourage work, and post-election we should stop giving free TV licences, winter fuel payments and bus passes to millionaires. People are worried about the cost of living. The government should get tough on privatised utilities such as water, energy and the railways, to squeeze down costs.

Last but not least, Cameron should take some Swedish lessons. Fredrik Reinfeldt, the Swedish PM, is a fellow conservative moderniser, and in 2010 led his party to its best result in 100 years. He outflanked the dominant left party, by focusing on jobs and low income tax relief, and so repositioning his moderates as "the new workers' party". This should be Cameron's ambition.