"We have no Eton to create the self-consciousness of a governing class," the American writer F. Scott Fitzgerald lamented 90 years ago as he compared the talented rulers of the British Empire with the mediocrities who then ran the United States.

This week, in Manchester, there are Tories quietly wishing that Britain had no Eton either. The polls are telling them that although the voters are sick of the Labour Government, they are not convinced that they want the Conservatives back. The Tories are not quite sure why the public has not learned to love them yet, but they wonder if class could prove to be their weakness.

In David Cameron they have their poshest leader since Sir Alec Douglas-Home led them to defeat in 1964. The question of class has dogged the Conservative leader all week since he was asked by Andrew Marr on Sunday morning whether it was true that he and his wife Samantha have a combined wealth of £30m.

Download the new Independent Premium app Sharing the full story, not just the headlines

Cameron's reply was predictably uninformative. "House prices change all the time. Our main asset is the house that we own in London," he said. He also confessed to being "desperately embarrassed" about a celebrated photograph taken when he was 19, showing he and Boris Johnson as members of Oxford University's elite Bullingdon Club. The tailcoats in which the young men were dressed cost £1,200 back in the mid-1980s.

When Samantha Cameron first entered the public eye, she too seemed embarrassed about her wealthy background. She claimed that she was brought up "near Scunthorpe". This was true, in a way. Her father, Sir Reginald Sheffield, owned Normandy Hall, a stately home set in 300 acres of north Lincolnshire countryside, and could trace his ancestors back 700 years. Samantha Sheffield's mother later married Viscount Astor, an Old-Etonian Tory peer who helped David Cameron in the early stages of his career. Lady Astor is co-owner of Oka, the upmarket mail-order furniture business.

Tonight, satellite television viewers will be treated to When Boris Met Dave, a fictionalised account of the adolescent years of Johnson and Cameron, set in the Brideshead Revisited world of Oxford University in the period just after Margaret Thatcher had crushed the unions, deregulated the City and allowed privileged young people to feel relaxed about their wealth.

Party spokesmen say they are not worried about this reminder of the silver-spoon beginnings of their two best-known public faces. "Everybody knows what their background is," a spokesman said yesterday. "David has never tried to hide it: he is obviously from a wealthy background. People will remember that Tony Blair was from a wealthy background, too. That did not affect Labour politically."

Tories also point out that the Labour Party tried this line of attack at the by-election in Crewe and Nantwich in May last year, when they branded the Conservative candidate, Edward Timpson, whose great-great-grandfather founded the Timpson shoe chain, a "Tory toff". The voters were not impressed, and Labour lost heavily.

Privately, members of the Shadow Cabinet admit that the class issue could flare up in a general election with more serious consequences for the Conservatives. The result in Crewe was the local electors' verdict on Gordon Brown and the Labour Government rather than a positive vote for Mr Timpson. In a general election, people might look more closely at the sort of people who are lining up for ministerial posts.

They will find yet more Old-Etonians including Oliver Letwin, who heads the Conservative research department, Sir George Young, a hereditary baronet recently brought back into front-line politics as Shadow leader of the Commons, and David Cameron's trusted chief of staff, Ed Llewellyn. Cameron's closest political ally, George Osborne, is another product of inherited wealth and a private education (at St Pauls in south-west London, the same school as the Shadow Culture minister, Ed Vaizey). Nick Herbert, the Shadow Environment Secretary, went to Haileybury in Herrtfordshire while Michael Gove, the Shadow Schools Secretary, was also at private school for part of his boyhood. It was with good reason that when David Cameron became leader four years ago, the Tory MP Nicholas Soames, grandson of Sir Winston Churchill, said jubilantly that the "natural order" had been restored.

This does not necessarily mean that they will grind the poor into the dust when they assume power. History suggests that the opposite is more likely. The Conservative Party was a gentler organisation when it was run by people who were born to rule, such as Harold Macmillan, than when it was taken over by people who had come up the hard way, like Margaret Thatcher and Norman Tebbit. Many on the right fear that David Cameron will try to compensate for his background by not being Tory enough.

But for most people, this is a political party which proposes to freeze the pay of public sector employees while making sure that people whose parents own homes worth up to £1m can inherit the lot, free of tax. The danger for the Conservatives is that voters who can barely stomach the thought of voting Labour again will look at the galaxy of clever ex-public schoolboys running the opposition – and conclude that they are a privileged bunch with no idea of what it is like to live off an ordinary working wage.