UKIP was never supposed to get this far. When Nigel Farage and his insurgent party won the European elections in May, the consensus in Westminster was that Ukip would fall off the radar as attention turned to the general election. But since then the party’s revolt has gathered pace. In the past three months alone Ukip has gained a new multi-millionaire donor and its first elected MP in Conservative defector Douglas Carswell. It has watched its membership swell to more than 40,000 and held a stable 15 per cent in the opinion polls. This is around twice the level of support it needs to inflict real damage on the main parties next May.

Ukip’s latest breakthrough in the Rochester and Strood by-election, where its new recruit Mark Reckless polled 42 per cent of the vote, marks a new turning point. The former Conservative MP finished eight points clear of the Tories, with a majority of almost 3,000. This might look convincing enough to encourage other disgruntled Tories to switch sides before the general election.

The result is significant for many reasons but the most important is demography. Rochester and Strood was never ideal territory for Ukip. Unlike other seats along England’s East Coast that have given Ukip its strongest levels of support, this Kent seat has more younger, middle-class and ethnic-minority voters. While it has pockets of deprivation and a struggling hospital, the seat is a fairly average place. When I ranked every seat in the country based on how many “Ukip-friendly” voters they had, Rochester and Strood emerged at number 271.

That Ukip can win in this kind of seat while faced with the full weight of the Conservative Party machine and David Cameron, who personally led his MPs into the seat promising to “throw the kitchen sink” at defeating Reckless, should ring alarm bells. While the result will inevitably fuel rumours about defections, it will also cast a long shadow over a much larger swathe of Conservative-held territory in southern England.

There are questions for Labour, too. How was a small challenger party able to finish comfortably ahead of the main party of opposition, in a by-election where voters said that the NHS was the most important issue — and in an area of the country that Labour used to control? Emily Thornberry MP’s fateful tweet yesterday, of a white van outside a modern terraced house festooned with St George’s cross flags, speaks volumes about Labour’s distance from its core working-class vote.

This election also shows how Ukip’s revolt is spreading in other ways. Researchers have long been pointing to Ukip’s appeal not only among disillusioned Right-wing Tories but also disgruntled blue-collar workers who used to support Labour. While Ukip continues to be portrayed as merely a second home for Eurosceptic Conservatives, since 2012 the party has shown an increasingly ability to recruit votes from across the political divide. In Rochester and Strood, polling by Lord Ashcroft shed light on the political backgrounds of Ukip voters: while 44 per cent of those who voted Conservative in 2010 planned to switch to Ukip, so too did 40 per cent of Labour voters and 23 per cent of Liberal Democrat voters. This looked more like a broad church than a Conservative Party in disguise.

The inability of the main parties to contain this revolt ultimately reflects a failure to understand its root cause. Ukip is not merely driven by protest politics. Nor is the party a by-product of the financial crisis, or a dislike of David Cameron. Rather, its appeal is rooted in much deeper divisions in British society that first opened in the 1970s and have been widening ever since. In short, Britain’s transformation has not affected all groups in the same way: some have felt left behind, cut adrift from a new mainstream consensus.

Ukip is strongest among older, working-class and white voters who earn around £24,000 each year, have few qualifications, were struggling long before the crisis and then got hit the hardest by austerity. On a range of economic issues these voters are closer to Labour than Right-wing Tories: they feel that ordinary workers are not getting their fair share and are openly hostile towards big corporations. But critically, they are also among the least likely of all voters to feel the recovery and to expect to feel it in the future, which partly explains why they are not rushing back to Conservatives talking robotically about the recovery.

It is these struggling voters who have fuelled Ukip’s rise since 2010 and made it the most working-class party in Britain since Michael Foot led the Labour Party. While they are insecure about their financial position, these voters are as much concerned about cultural issues as economics. The left-behind share a fundamentally different set of values from a London-centric, university-educated and more socially liberal middle-class, which they see as representing everything that is wrong with modern Britain.

These are not new views. Many were instinctively hostile to immigration, the EU and Westminster politicians long before Nigel Farage showed up. He simply identified the divides and mobilised them back into politics. Nor does he follow the main politicians in talking to these voters in narrow transactional terms. Again and again, the main parties attempt to win Ukip voters back by setting out the economic case for migration and European integration. But these voters care less about how migrants or the EU contribute to our national wealth than how this social change is impacting on a perceived set of values, way of life and identity that they cherish. Their concerns are diffuse and abstract; they cannot be revolved through a policy tweak or a populist speech.

Don’t think about Ukip as a political party. Think of its revolt as a symptom of much deeper conflicts in modern Britain: between those who have the resources to adapt amid rapid social change and those who feel left behind — and intensely anxious about this change.

Matthew Goodwin is Associate Professor in Politics at the University of Nottingham and co-author of Revolt on the Right: Explaining Public Support for the Radical Right in Britain (Routledge)