Back in September, Boris Johnson had a meeting with some of the 21 Tory MPs who were about to lose the party whip over their opposition to a no-deal Brexit. He was warned that if he was prepared to risk the hardest of Brexits, he might lose crucial parliamentary seats – reliably Tory places where a majority backed remain. One of the people present was Anne Milton, the MP for Guildford, the large Surrey town won by the Tories in 2017 with 55% of the vote and a 17,000 majority. She says she expressed her fears of defeat, only to have them thrown back at her. “If Guildford’s lost, it’s lost,” said the prime minister, and that was that. Two months later, Milton is standing as an independent, Guildford may well be a four-way marginal – and one of the election’s most fascinating stories is rolling on.

As people are priced out of London, they are bringing a more cosmopolitan culture with them

In the face of all the noise about “Workington Man” and the purported significance of leave-voting seats in the north of England, this particular subplot has been overlooked. It is about the extinguishing of David Cameron’s Tory “modernisation” project – and the growing disconnect between voters in affluent Conservative heartlands and the ludicrous ideology of Jacob Rees-Mogg and the European Research Group.

While the Tories have been transforming themselves into the nationalist, Pooterish party of hard Brexit, essentially cosmopolitan attitudes – open, internationalist – have been seeping into the formerly Conservative areas of our cities, and the suburbs and towns around them.

Some of this was evident in 2017, in the seats lost by the Tories to both Labour and the Liberal Democrats: Bath, Kingston and Surbiton, Battersea, Enfield Southgate. The same story is reflected in the weakening Conservative presence around Manchester and Bristol – and Lib Dem targets such as St Albans in Hertfordshire. But perhaps the most interesting story lies in Surrey, that traditional signifier of true-blue Tory England.

Until recently, the county had 11 Tory MPs. But by the time the election was called, the Conservatives had expelled Anne Milton, Philip Hammond and Sam Gyimah (now a Lib Dem). In local elections this May, the Tories lost 117 council seats in Surrey and control of four local authorities. Among Surrey’s eight remaining Tory MPs are such Brexiters as Dominic Raab, Michael Gove and Kwasi Kwarteng. But in the borough of Guildford, remain got 56.2% in the 2016 referendum; in Woking, 56.2%; in Elmbridge, which includes Raab’s seat of Esher and Walton, nearly 60%.

‘If Guildford’s lost, it’s lost,’ Anne Milton said the prime minister told her. Photograph: Steve Meddle/Rex/Shutterstock

I went to Guildford last week. Of late, it may have seemed rock-solid Tory. But way back in 2001, the Lib Dems took it with a majority of 538 – which, combined with an uptick in 2017 and recent gains in local elections, seems to have convinced them they can win again. Two years ago, Labour got close to a not-unreasonable 20% of the vote. Obviously, the Conservatives are still a force to be reckoned with. But now that Milton has been thrown out of the Conservatives and is running a campaign so far based on “a smartphone and a laptop”, there is a clear sense of no one having a clue what will happen: it’s the election in microcosm.

When I talked to people in the centre of town and observed some Lib Dem canvassing in the upscale neighbourhood of Onslow, one thing was striking: as well as a smattering of staunch and sometimes angry Brexit supporters, there were many people who said that post-2016 Conservatism had left them feeling abandoned. “Labour and the Tories are too extreme,” said one man, who up to now had been a loyal Conservative. “I want a centrist party.” These days, “centrist” is usually thrown around as an insult; he said it with a sense of pained loss. A little earlier, I had met a churchgoing couple in their 60s – he a geologist, she a self-described housewife – who had often voted Conservative down the years, but were also switching. “It feels like we have to vote on Brexit, and we’re remainers,” they said.

It is easy to still think of the home counties in terms of a quintessentially suburban mindset – prim, reactionary, moralistic and deeply Conservative, with both small and large Cs. The massively popular 1970s sitcom Terry and June, a camped-up glimpse of a couple trying and failing to live the perfect middle-class life, was set in Purley, on the outer London/Surrey borders. In the songs of those ageless sons of Surrey, the Jam, there are portraits of characters who fit much the same picture: Mr Clean (“Getting pissed at the annual office do/A smart blue suit, and you went to Cambridge, too”), and Smithers-Jones, the hapless commuter who “stops off at the corner shop to buy the Times”, turns up to work as usual and suddenly gets the sack.

Immerse yourself in those songs, and you hear a world that endures, but is also in the midst of change. The population of Surrey is becoming more ethnically diverse, and as people are priced out of London, they are bringing a more cosmopolitan culture with them. On Guildford’s north side, the University of Surrey draws in a multitude of people from overseas, and lots of students from London. The town’s support for remain reflects a blunt economic question: if your livelihood is dependent on the capital’s booming economy, why would you vote to leave the EU?

And so to another question: what does post-Cameron, Rees-Moggist, Brexit-or-bust Conservatism really have to say to people in what we still know as the commuter belt? Perhaps the usual Tory warnings about Labour’s tax plans will cut through; maybe dislike and mistrust of Jeremy Corbyn will do its work. But through the seemingly endless Tory mishaps of the last couple of weeks, I started to sense a party that well knows that the ground under its feet is shifting. When parties start to become accident-prone, there are usually deep reasons why. The sense of Conservatism weakening in its own heartlands is surely some of the explanation.

This story might not necessarily be about heavy losses in five weeks’ time. What is happening may be moving more slowly and unpredictably than that. But any Conservative with their eye on the Tories’ long-term prospects would be well advised to think very hard. Parts of England are drifting away from a political tradition that once represented them as a matter of instinct, but that now seems to be lost in a murk of nostalgia and Brexit dogma. And whatever the daily hurly-burly of the election campaign, it is these things that tend to decide whether or not political parties have a future.

• John Harris is a Guardian columnist