This week David Cameron celebrates his fourth anniversary as Tory leader and is now probably just 22 weeks from becoming Prime Minister. True, the polls last week showed a Tory wobble, although the latest YouGov poll in marginal seats suggests that the Tories are at last scoring where it matters – in the marginal seats in the Midlands and the North.

Yet as Mr Cameron stands on the threshold of power it looks as though the issues of Tory toffery and class will, once again, raise their ugly head as the pre-election campaign threatens to turn nasty. My first instinct, as a secondary modern school 11-plus failure, should be – according to well established Labour theology – to reach for the ruler to measure the size of the chip on my shoulder when such stories surface. But when I stood in Scunthorpe as the successful Tory candidate in 1979 I found little evidence that my supposed "asset" of working class authenticity played any significant part in my victory. A smear campaign claiming I was "thick" was still played against me by my opponents. More likely they were actually playing the anti-gay card which was their normal method of smearing me – the Lib Dems were usually the worst offenders.

So their call for the multi-millionaire Zac Goldsmith – challenging them in Richmond Park – to give up his non-dom tax status suggests that many Tory candidates may face smear campaigns from their opponents – whatever their backgrounds. I suspect, however, that if elected, Mr Goldsmith's wealth is likely to bolster his independence of thought on environmental issues and will endear him to those of us who look forward to him being a free spirit impervious to the petty demands of his whips' office. I recall Sir Peter Tapsell congratulating me on becoming a government whip, reminding me rather menacingly that he was a very wealthy man, did not want a government job and nothing in the way of foreign junkets. "There is nothing I want and nothing you can threaten me with," he boomed. When it came to the Maastricht debates he was utterly beyond the patronage of the whips' office and all the better an MP as a consequence.

Download the new Independent Premium app Sharing the full story, not just the headlines

In the heat of the election battle, I suspect the dominant feature will still be the economy. But the noises off, regarding the character and education of the next Tory Parliamentary party, may cause Labour to play the old school tie card. When the Tories were last led by Old Etonians Harold Macmillan and Alec Douglas-Home, in 1963, 20 per cent of their MPs (80) were also from Eton. For the record, the number of Old Etonians, including Mr Cameron, on the Tory benches is likely to rise from 15 to 18 – less than 6 per cent of the 300-plus Tory MPs likely to be elected. But, according to the Tory blogger Iain Dale, 15.75 per cent of daily and Sunday newspaper political editors are Old Etonians.

If anything, it is the prominence of double-barrelled names among new candidates that seems most to trouble Mr Cameron. According to recent newspaper reports, the Tory leader suggested, jokingly, that it would be better if the Tory candidate for Somerton and Frome (a Lib Dem marginal) changed her name from Annunziata Rees-Mogg to Nancy Mogg. Rightly she told Mr Cameron she rather liked her name. I met her last week and her posh name could not hide the fact that she is one of the most approachable of women candidates and has a reputation for being a brilliant local candidate. Her brother, Jacob, (Old Etonian) who is likely to gain the neighbouring seat of North-East Somerset from Labour, is certainly no "Jake Mogg" but his original mind will make him a whips' nightmare and an outstanding MP. The Tory candidate for Brighton Kemptown, once known as Simon Radford-Kirby, has apparently succumbed and now calls himself plain Simon Kirby. Yet Mr Kirby is not posh: he was brought up in a council house. On the other hand, what are we to make of the Tory black farmer, Wilfred Jones, likely to win the newly created Chippenham seat, who has changed his name to Wilfred Emmanuel-Jones?

Labour and the Lib Dems should think carefully before playing the Tory toff card. Last year's Crewe and Nantwich by-election ended in disaster for Labour when their campaign targeted the alleged wealth and educational background of Edward Timpson. Top hatted and tailed Labour activists trailing after Mr Timpson merely allowed Eric Pickles, the Tory chairman, to make the point that Labour were the party who canvassed in fancy dress.

The Tory leadership is (unduly) sensitive to such criticisms. That is why Mr Cameron has invested so much personally in supervising candidate selection procedures. But Labour and Lib Dems play the politics of envy at their peril.

mrbrown@talktalk.net