At last, it seems to be hitting home. The chatter about the result of the Eastleigh byelection may have its overheated aspects, but at its heart is something incontestable. It is what some people would call a crisis of political representation, highlighted by Ukip's 28% of the poll and second placing, and the cacophony of noise in response to its success.

Over the weekend, so many politicians and pundits told the same story that by Saturday evening it had become a cliche: a tale of an out-of-touch political class, and an electorate that will punish them at any opportunity. That this view of things was voiced by such latter-day Wat Tylers as the former Daily Telegraph editor Charles Moore and senior rightwinger Bernard Jenkin only confirms how deep the problem is, and how much it eats away at the soul of the Tories. As well it should – for this is essentially a story about conservatism, with a small and a large "c", and the fact that in England, no mainstream party truly understands or gives convincing voice to it.

It dates back at least 30 years. From the early 1980s onwards, Margaret Thatcher and her governments embedded a new notion in the collective Tory mind, and British politics more widely: that politicians should be judged by their radicalism and obstinacy. Self-evidently, this was not Conservatism as anyone had previously understood it – but up until the poll tax saw boldness curdling into hubris, the party and its wider constituency were in almost full support. This was because grim times seemed to demand drastic answers, and because the Thatcherites' mouldbreaking economics were intertwined with their social conservatism.

Meanwhile, serial defeats for the Labour party eventually led to the arrival of the cult of the so-called modernisers, pledged to force their party to swallow the fact that Thatcher had changed the country for keeps. Though Gordon Brown eventually spurned this next aspect of their credo, Tony Blair and his followers also came to be believers in the permanent radicalism that had so gripped the Thatcherites. "Reform" was their watchword and they had one new article of faith: that the best proof of any leader's bona fides was the habit of loudly defining themselves against their own side. Eventually, this became almost pathological, as swaths of the party, the unions – and, by extension, millions of voters – were decried as hopeless throwbacks.

By 2005, the game was almost up: in the aftermath of the Iraq war, let us not forget, Blair somehow won an election in which his support was down to not much more than a fifth of the electorate.

By then, it was the Tories' turn to be taken over by modernisers, passionately in thrall to Blair and his disciples' example. David Cameron's early stabs at offending Tory traditionalists – hugging huskies and hoodies, the contemptible W10 wind turbine – were barely serious, but their effect still lingers, and the same approach was seen when he picked a fight over same-sex marriage.

Most people on the left applauded that move, and rightly so, but that is not the point: in the eyes of his detractors, here was more proof of a clique prone to see their own side as a source of frustration and annoyance. This is something Cameron reportedly makes as much personal as political. His remains a politics rooted in upscale capital postcodes rather than the shires and suburbs: hyperactive and reform-crazed, and characterised by a Blairesque belief that any worthwhile prime minister must notch up plenty of foreign entanglements .It is largely rightwing, perhaps, but hardly conservative.

And in that sense, it bumps up against a huge and inconvenient fact. Albeit with a small rather than a large "c", conservative still describes millions of people. They do not like the establishment's tendency to piety (or, if we must, political correctness), and fear that it has now gripped most of Westminster.

Many suspect that the politics of climate change amounts to so much hysteria. They find the recent experience of immigration troubling – not because they are racists, but because they have justified worries about whether our social fabric can cope. The EU does not annoy them quite as much as some people think, but its distant authority and relevance to immigration makes them open to the idea that we may be best off leaving. And yes, many of them are unsure about the idea of same-sex marriage – not because they justify all those amped-up warnings about "bigots", but because it was a radical change to an enduring institution, and such things always cause some people unease.

To acknowledge all this is not to endorse it: they are not my tribe. But within their politics, there are elements that are traceable to the left rather than the right: an enduring belief in the NHS, a common conviction that the railways would be best off renationalised.

Their views on so-called welfare can seem punitive, but they may yet be rattled by such injustices as the spare bedroom tax, and what the government is doing to disabled people. Besides, though many read the Daily Mail, they do not share the apocalyptic views of, say, its renowned columnist Melanie Phillips: boiled down, their take on the world amounts to a gentle though occasionally tetchy scepticism. It is as much about broad values as anything specific: anti-metropolitanism, a profound dislike of hype and cant, a belief that governments should see to the home front before they fret about anything abroad.

In some places, none of this is any great bar to voting Labour. Indeed, in Wales that party is to some extent a conservative institution, there to keep Westminster's market-based meddling at bay, and cling on to what remains of the post-1945 settlement. In Scotland, part of Alex Salmond's triumph has been his contempt for Blair and his legacy, the SNP's resistance to London-born ideas of "reform", and their 58-year-old leader's embodiment of cultural continuity rather than gimmicky change. But in England, conservatism's story remains bound up with the Conservative party – and here, Cameron is found wanting, while Nigel Farage has enough craftiness and political leeway to make hay.

Panicked, the Tory leader assures us there will be no "lurch to the right", just as Blair used to follow electoral wobbles by assuring the country that he wanted nothing to do with the left. Cameron claims to empathise with people who, in the recent past, "wanted to talk about Britain being great again" but were "made to feel nostalgic and old-fashioned". But the key aspect of his predicament is impossible to get around: it is not what he does, but who he is. Whether they warrant a big or a small "c", most conservatives want to be led by one of their own – and in any decently functioning democracy, that is surely the least they deserve.