In a seemingly endless season of Tory nightmares, this week looks set to mark the most dreadful phase so far. The Conservatives are about to endure a set of elections that they never thought they would face. Only four years ago, the party won a general election; now, there is talk of them finishing fifth, or even sixth. With every Tory moan of pain, Nigel Farage’s nicotine grin grows ever larger. And out in the country, there is an overlooked Conservative crisis: one bound up not with the part of the population that voted for Brexit, but with the liberal, pro-remain swathe of the country without whom the future of Conservatism looks bleak indeed.

I come from somewhere still understood as one of the most Tory places there is. Wilmslow, in Cheshire, has a population of 25,000 and is a dormitory town on the southern edge of the sprawl around Manchester. Part of the Tatton constituency, it was once represented by George Osborne, and these days is the adopted home of the zealous Brexiteer Esther McVey. Though slightly more mixed class-wise than its reputation might suggest, it remains a byword for suburban affluence, and McVey sits on a majority of 15,000. But in the referendum of 2016, Wilmslow was part of a wider Tory-supporting area that voted 54% for remain.

A politics of nostalgia and nastiness is the complete opposite of what these voters look for

I was back last week, trying to get a sense of how the place has changed since I grew up there. As central Manchester has boomed, Wilmslow seems to have let go of the old idea that it is somewhere completely separate from a once depressed and disadvantaged city, and has happily become Manchester’s adjunct, both culturally and economically. The accents in the cafes are more Mancunian; the population is no longer uniformly white; the regular farmers’ markets suggest that its shopping habits have taken on a tinge of green. At this month’s local elections, moreover, my hometown lost three of its four Tory councillors to a new independent grouping, among whose prime movers is one of my former comrades from Wilmslow Labour party (once a small and noble grouping, but that is another story) .

Seven miles away, all the councillors in the once solidly Tory town of Altrincham are now Green, part of a bigger story of realignment across the suburban borough of Trafford that has led to the Tories losing control to Labour. And this spectacle of Tory retreat is hardly unique to the north-west. Look at how power has suddenly changed hands in an array of local councils, and you see the same picture: in areas where there was a majority for staying in the EU, the Tories suffered serious losses. This happened in such places as Bath and North East Somerset, Cotswold (the local government area centred on Cirencester), South Oxfordshire, Guildford and St Albans. There was a sense of much the same thing in some of the places the Tories lost to Labour at the 2017 general election: Croydon Central, Bristol North West, Cardiff North, the unlikely socialist hotbed that is Kensington.

‘When people behold the weird, cultish Conservatism of Mark Francois [pictured], Steve Baker and Jacob Rees-Mogg, what do they think?’ Photograph: George Cracknell Wright/REX/Shutterstock

Each time I go home, the cultural transformations that sit under all this become clearer. The modern stereotype of middle-classness – regular trips to Sainsbury’s, dutiful recycling and visits to the gym – is very different from the stuffy, hidebound ideal that prevailed a few decades ago. The expansion of higher education and the fact that university tends to make people more liberal looks set to deepen these changes even further.

At the same time, even apparent success now comes with a large dash of anxiety and insecurity, something that really kicks in among the younger age range. For sure, the English middle-class might still be attracted to the politics of low taxes, ever-increasing house prices and the overriding imperatives of “business”, but its essential attitudes are now much more complex and category-defying than they were through most of the 20th century.

The fact that David Cameron and George Osborne wanted to “modernise” Conservatism was a recognition of this. Now, you wonder: when millions switch on their TVs or look at their phones and behold the weird, cultish Conservatism of Mark Francois, Steve Baker and Jacob Rees-Mogg, what do they think? This is not to suggest that bourgeois England is about to run into the arms of Jeremy Corbyn, but rather to highlight that the intersection between supporting remain and having voted Tory in the recent past is a lot larger than some people would like to think. A politics of nostalgia and nastiness is the complete opposite of what these voters look for.

On Thursday, the Brexit party is predicted to attract the single biggest share of votes, while the Tories’ performance will be catastrophic. There seems little doubt that the Tories’ internal tensions will then go nuclear – and the leadership election that has already started will decisively step up. Listen to the mood music: the Spectator magazine is warning of Tory voters “defecting en masse” and claiming that Boris Johnson is best suited to “taking on Farage and putting a smile back on the faces of Tory members”; former minister Crispin Blunt is already talking about electoral pacts with the Brexit party.

Everyone knows what will happen, sooner or later. Given the ageing profile of the Tory membership and its attachment to hard Brexit, the party will move even further to the right. Put another way, it will fix its collective sights on the kind of old, zealous, implacably parochial people who have been turning up to all those Farage rallies, while all the demographics continue to move in the opposite direction.

Brexit is not the only factor at work here. Quite rightly, the decade-long conversation about austerity has tended to focus on people who have been least able to adapt and defend themselves. But there is a middle-class story here too. On this score, I think of the sage words of the historian Ross McKibbin, written in 1999: “The middle classes make more use of the NHS, public transport, public libraries, local swimming pools, public parks and their right to state welfare than anyone else.” Twenty years on, their councils have no money left, and the public realm is decaying in front of their eyes.

It increasingly feels to me as if something is afoot akin to the massed waking-up to the state of the country that happened in 1997, only this time there is no universally popular political force that can mop up all the anxiety, as Tony Blair’s Labour party did. Instead, middle-class angst adds to the inescapable sense that just about everything in Britain is riven with unfixable cracks.

There is a rising Tory fantasy about the party’s immediate post-May future, which seems to be based around victory in a general election, a new leader going back to Brussels full of swagger – and, if need be, Britain stoically going it alone. If that happens, the revolutionary, Brexit-or-nothing school of modern Conservatism will reach peak arrogance, thinking the ghost of the blessed Margaret Thatcher is cheering everything on, and that its moment of destiny has arrived. The truth is the exact opposite: whatever its delusions, hyper-Toryism – and, by extension, the Conservative party itself – is sliding into a elemental crisis from which it may never recover.

• John Harris is a Guardian columnist