Seventeen miles south of Bath and a half-hour drive from Glastonbury, the Somerset town of Frome is one of those English settlements that rather suggests the country in microcosm. Bits of the townscape have a pinched, austere aspect, though close by there are cafes and clothes shops brimming with chichi affluence – walking around the place, you occasionally get the sense that you can traverse the entire social spectrum in less than 50 yards.

The town forms the core of the constituency of Somerton and Frome – number 11 on the Conservatives' target list, and currently held by the Liberal Democrats with a majority of 812. On a swing of only 0.56%, it would fall to the Tories and be represented by a new MP: 30-year-old Annunziata Rees-Mogg, daughter of the life peer and establishment mainstay William Rees-Mogg and sister of the Eton-educated Jacob, who just happens to be favourite to win the neighbouring seat of North East Somerset.

Thus far, Rees-Mogg has crashed into the national headlines only once: last week, when, with the rising noise of class warfare coming from Westminster, there were reports of David Cameron bemoaning the number of Conservative candidates with double-barrelled surnames, and suggesting – as a joke, his people later claimed – she change her name to the altogether more approachable "Nancy Mogg".

Frome, then, seems an appropriate setting for a thoroughly non-scientific exercise: trudging the streets with an A4 picture of Cameron, and attempting to find out what people make of our probable next prime minister. Who is he? What words does he bring to mind? And what might he do with all that power?

"He's like the young Tony Blair," says 34-year-old Sean Kirtley, whom I grab as he leaves a kitchen shop with his young daughter. "Politicians are all very smooth these days, aren't they? Would I vote for him? I haven't made my mind up. But he's obviously talented. And he's got more charisma than the man in power at the moment."

Over the next hour or so, I make my way through the uncertain tangle of feelings that will decide the next election. People talk about Cameron as an assured, on-the-up kind of figure – and routinely contrast his skills with a sitting prime minister who, true to recent reports, prompts a kind of sad sympathy. The more malign stuff I hear about the Tory leader includes "no substance", "too much of an opportunist", and "he's trying to get across that he's the common man, but he isn't". Those with a more positive opinion mention his credentials as a "family man", his air of winning confidence – and I have no great difficulty finding people who think he should soon be in 10 Downing Street.

"I hope he gets in," says a thirtysomething mother who talks for England but won't give me her name. She says she voted Liberal Democrat in 2005, but having opted for the Tories in the past, she's now going to revive the habit. "He comes across as an honest family man. I work in education, and I hope he'll make some changes. I think education's underfunded, but I also think there's far too much bureaucracy and paper-pushing. I know we're in massive debt, and he's probably going to put taxes up. I don't want to pay more tax, but I'd rather pay tax that's well spent. At the moment, it isn't."

At the top of the cobbled Catherine Hill, however, I get a rather more swingeing verdict. The Little Red Cafe is the kind of small business that would surely cause your average Cameroon to swoon – all beautiful coffee, home-made cakes and retro-look fittings – but Liz Andrew, the co-proprietor, may not be their easiest quarry. Her answers, in fact, would gladden those Labour politicians who are currently trying to portray Cameron as Lord Snooty-gone-Thatcherite.

"What do I think?" she wonders, half-glaring at Cameron's portrait. "Slimeball. He represents the kind of smarmy upper-middle-classness I don't like, really. I've never been a fan. I wouldn't put my trust in him, put it that way."

I wonder: can she think of one story that sums up what puts her off?

"Probably the cycling to work thing," she says. "You know, with his limo driver behind him."

As is now a Westminster cliche, the occasion in spring 2006 when the Daily Mirror caught Cameron cycling to work while his chauffeur brought up the rear with that day's paperwork has long been established as one of his biggest mistakes. More than three years on, it still haunts him, and the fact that it is fixed in some people's minds as a byword for chicanery and insincerity crystallises another modern political commonplace: the Tory leader's supposed failure to "seal the deal".

November was not exactly a glorious month for the Tories. First, Cameron bowed to the inevitable and binned his supposedly "cast-iron guarantee" of a referendum on the Lisbon treaty – to howls of protest from the Eurosceptic right. At around the same time, some voices suggested that the Sun's shrill noises-off about Gordon Brown's letter to the mother of a dead Afghanistan veteran had tainted the Tories by association.

There were also ongoing mutterings about how the new Conservative emphasis on cuts and austerity was backfiring, those revelations about Zac Goldsmith's non-dom status, and Labour's attacks on Tory plans to effectively raise the threshold of inheritance tax to a cool £2m. Small wonder, perhaps, that seven of last month's 10 British opinion polls gave the Conservatives less than 40% – the figure they need if they're to achieve a majority of Westminster seats.

Two weeks ago, moreover, the Observer ran an Ipsos MORI poll that put only six points between Labour and the Tories, and it all went off: endless media chatter about the rising chances of a hung parliament, and the possible dashing of all kinds of Conservative hopes.

And so to the big question. In the summer of last year, when Brown's woes over the abolition of the 10p tax rate fed into Tory poll leads of more than 20 percentage points, the great Conservative dream of a Blair-esque landslide seemed within their reach. Why has that prospect now seemingly disappeared?

When I speak to Ipsos MORI's chief executive, Ben Page, he says he was surprised by their last poll, and is at pains to remind me of Cameron's enviable personal approval ratings – but eventually he explains some of what might be eating away at the Tories' popularity.

First, Page mentions rising optimism about the state of the economy – according to Ipsos MORI's research, the gap between positive and negative feelings about the economy's prospects for next year is at its highest level for 12 years. Next, he focuses on a general unease that seems to distance the Tories from some of their potential supporters: "I think there are still issues about the fact they haven't been tested in office. There are lots of people who don't like Old Etonians, though a lot of them are concentrated in Labour seats. And there's maybe a nagging anxiety about public services. None of those things are killer factors, but there's a slight queasiness about the fact that 'These posh people don't really understand real lives, and they're coming in to run to the country.'"

Up at the University Of Strathclyde, the psephologist John Curtice talks about the share of people who describe themselves as "Tory identifiers" – not just Conservative voters but, in their own eyes, essentially Tory people. As things stand, the numbers of Tory and Labour identifiers seem to be pretty much equal, meaning that the Conservative poll lead is classically "soft" – made up of people who, as he puts it, "have at least the potential to disappear".

Curtice continues: "The reason the Conservative party looks to be in a strong position is, frankly, not because they're doing particularly well. It's just that the Labour party is doing so badly. In terms of shares of the vote, no government at this stage of a parliament has been as unpopular as this one. The fact that Conservatives often have a double-digit lead is simply down to them being on around 40 and the Labour party coming in under 30. And 40%, by historical standards, is nothing."

Cameron, as Curtice acknowledges, has problems that Tony Blair never experienced. Most fundamentally, unlike New Labour, the Tories are faced with an electoral system with an in-built bias against them: it must seem particularly cruel that Labour got a majority of seats in 2005 on a mere 35.3% of the vote. Consider also the seismic impact of the expenses crisis, and how it has squashed the prospect of any vast outpouring of enthusiasm for mainstream politicians. In 1997, Labour was able to wage war on so-called Tory "sleaze" and trade on the expectation of politics somehow being redeemed, which contrasts sharply with the across-the-board cynicism that represents one of Cameron's biggest obstacles.

And then there is the problem posed by all those increasingly popular "other" options, in particular the UK Independence party. When I call Nigel Farage – who recently stepped down as his party's leader, but remains a zealous Ukip mouthpiece – he does exactly what you'd expect, revelling in his party's current status as troublemakers to Cameron's right.

"I spent some time last weekend in Norfolk," Farage tells me. "I stayed at a country house, and went shooting with seven land-owning, farming-stroke-business Norfolk types. One of the people I shot with gives the party a very large sum of money every year. That's just one segment of England, but it's what you would call classic Tory territory.

"For them, the breach of the promise on a treaty referendum is pretty fundamental stuff. They're also becoming increasingly uncomfortable about the whole issue of global warming. Cameron's encouragement and support for wind farm projects, a belief in increasing green taxes – well, that's the second area where they find themselves at total variance to him. The third is tax, where the acceptance of the 50p band links into the idea that there's no real difference between him and Labour. The fourth I would go with, which came through very strongly in Norfolk, was education: the fact that Cameron has turned his back on the principle of selective education."

He finishes with a menacing flourish: "Over the last few years, there are Conservative sources who have said the Ukip intervention at the last general election cost the Tories 28 seats. There are signs that with Ukip where it is in the opinion polls today, we might cost them 50. And that could mean Cameron will not get a majority."

If Farage sounds characteristically full of himself, it may be some token of these strange political times that you do not have to try too hard to find a similar fighting spirit among Labour party insiders. For all Labour's miseries, some pizzazz has returned to their campaigning, as seen in last month's online wheeze, whereby Cameron and Osborne were done up as The X Factor's joke-cum-phenomenon John and Edward, with the words: "You won't be laughing if they win." The idea came from Labour's ad people at Saatchi & Saatchi – according to one Downing Street insider, approving it "was a no-brainer; there was no research involved, but even we were surprised at how much it took off".

There is, of course, a you-would-say-that element to Labour people's views of Cameron's alleged weaknesses – but it's hard to doubt at least some of what they say. Their essential charge is that many of the undecided voters Cameron needs to seduce are not convinced the Tories have undergone any kind of transformation. This chimes with a recent Populus poll, in which only 28% of people agreed the party has "really changed".

There is also, they claim, a palpable unease about the Tory leader. "People kind of like Cameron; in lots of ways, his scores would be pleasing," says Deborah Mattinson, Gordon Brown's personal pollster. "But underlying that is a sort of scepticism. Put it this way: the car and bike thing really sticks. Even now, I rarely do a focus group where that isn't mentioned spontaneously by somebody. It's one of those things that's stuck, and the reason is it speaks to a truth that people identify about him. There's something about him that doesn't quite ring true. That's very clear, and he's never managed to make it go away."

On the proviso that the observation is completely off-the-record, one Labour high-up tells me that Cameron's self-appointed role as the supposed heir to Blair may be working against him: "When people kind of fell in love with Blair, they hadn't really seen a politician like that before. Cameron is positioning himself in quite a similar way, and you know what? They've seen it before, and in the end, they didn't like it. It became very toxic, and I think there's a halo effect from that which is personally contaminating for Cameron."

Then there is this season's great political hoo-ha: class, the Tory leader's elevated background, and the fact that Labour has lately decided to start talking about it. There may be more of an appetite for this tactic than some people would like to think: according to last week's Sunday Times, 47% of people polled agreed with the assertion that Cameron is "too wealthy and privileged to represent ordinary people", and 52% endorsed the idea that the Tories are "still the party of the rich".

At the weekend, the privately educated Alistair Darling and the work and pensions secretary Yvette Cooper attempted to ratchet down the political attacks focusing on Cameron's Eton schooling, which points up an interesting unease among most of the Labour people I talk to. There again, echoing Ipsos MORI's Ben Page, one Labour pollster I speak to says that among the voters they meet, Cameron's background is undoubtedly an issue: "What it plays to is, 'Are they in touch with me? I'm a mum living in Harlow, working for £7.50 an hour and getting my kids to the childminder – what does he know about my life?'"

What's perhaps most telling is that it is not just archetypal lefties who point to the political salience of all this. "We did really heavy-duty polling over the summer," says Chris Fox, the Liberal Democrats' director of policy and communications. "One of the things we did was take 20 or 30 people in the south-west who said they were thinking about voting Conservative, and spent four hours with them. The overriding thing that came out was even people who were thinking about voting for them weren't doing it because of any real endorsement of their values. And as soon as you got them to pick imagery for the Tory party, it was all about class, looking after one part of society, and all those things. That's in the south-west; further north, it becomes even more ingrained."

So what can the Tories do? "It would be much better, psychologically and in campaigning terms, to be above 40%," says a high-ranking member of the shadow cabinet and Cameron ally. "But you can spend forever checking your pulse. The important thing is to be match-fit."

Among other things, he tells me that the Tories should be worrying far more about "aspirational centrists" than disaffected voters on the right, and agrees that with six months or so until the general election, there is much work to do on the kind of clear and punchy message that New Labour managed 12 years ago.

Having said that, he agrees a Tory version of 1997 would be unthinkable, pointing to the expenses crisis – and the damage done to the Westminster elite by the financial crash and recession – as the key reason why the outcome of the next election will feel altogether more equivocal. "It's very, very rare that you've had an opposition leader who has exercised the kind of supremacy that Blair did. The crucial thing there was, while the Tories were deeply tarnished and compromised, the opposition was fresh and untainted. The problem now is that all politicians are tarnished and compromised, and therefore the idea that you would invest hope in any politician is a lot more difficult."

The essential message coming out of the highest Tory circles is simple enough: in such hard and doom-laden times, you bin your expectations of a landslide, assume a much more realistic view of the world, and understand that only power will allow you to seal any kind of deal. Comparing an Old Etonian with British history's most famous grocer's daughter isn't the easiest business, but there are Conservative voices who point to Margaret Thatcher's uncertain place in the public's affections as leader of the opposition, and hold out the idea that Cameron and his people may be able to approximate the route that eventually led to her dominance of politics in the 1980s.

Still, there is a twitchiness around the Tory leadership – partly reflected in the way Cameron seems to be hyperactively bouncing all over the political spectrum, and from issue to issue. A fortnight ago, he was endorsing the Tory guru Phillip Blond's expansive vision of "Red Toryism", only to then expend no little energy bemoaning the bog-standard Daily Mail straw man that is Health and Safety. Towards the end of November, the Tories went on the offensive over the decline of marriage; yesterday, Cameron spoke to the single-parent pressure group Gingerbread. Underneath just about everything they do, perhaps, is an unresolved personality-split between their cuddly "progressive" side, and the part of them that cannot let go of the crabby, moralistic politics that are still threaded into the Tories' DNA. The upshot: none of the choreographed consistency that New Labour showed as they glided towards power in the mid-1990s.

So, the Conservatives may well take Somerton and Frome, and Crawley, and Harlow, and Cardiff North, and plenty of their other target seats, but even some of their own people think they might not get a majority at Westminster – at which point, we will plunge into a scenario we have not seen in well over 30 years.

Last week, I asked a high-ranking shadow minister what they thought was the most likely outcome of the next election. "A minority Conservative government," he told me. "But then again, I'm the most pessimistic person I know."