Last week I spent four days in Milton Keynes, the Buckinghamshire new town that sits in the English imagination as a byword for modernist architecture, endless roundabouts, and the fact that many of us still think that anything remotely futuristic is best sniggered at. The Conservatives’ conference provided the mood music; in between nights spent in a short-let, new-build house in the neighbourhood of Bletchley, I drove and wandered around business districts and ever-expanding housing developments, trying to get a sense of where the country has arrived.

Contrary to the received idea of the place as somewhere strange and almost unique, Milton Keynes reflects English politics in imperfect microcosm. Though it has two Tory MPs, 2017 saw its voters split pretty evenly between the Conservatives and Labour. The local electorate narrowly voted in favour of leaving the EU by a margin that matched leave and remain’s shares in the UK as a whole. The population reflects the diversity of modern Britain, and there are depressingly familiar signs of inequality: child poverty rates across its council wards range from just over 6% to about 40%, and the centre of town is smattered with those grimly familiar clumps of tents in which homeless people sleep.

A discourse of bitter partisanship, on both right and left, is leaving a lot of people cold

But compared with many of the areas that now symbolise the Brexit moment, one thing is missing: the weight of history. In lots of places I visit, the ghosts of a lost industrial past are impossible to escape. By contrast, the residents of Milton Keynes seem to live with their minds mostly focused on the present and future, a condition that defines much more of the country than some people think. Picture the new (or newish) private housing that now rings most towns and cities, and is often plonked amid distribution centres: a new country has been taking shape here, and it demands attention.

Play Video 15:56 Anywhere but Westminster | We must deliver: Brexit, Johnson and the robots of Milton Keynes – video

The absence of history in these places feeds into something else that was noticeable as soon as I started talking to people in Milton Keynes: surprisingly little evidence of the supposedly all-pervading polarisation and enmity – and burgeoning “culture war” – that too much of the media now sees as defining public opinion, something I wrote about a couple of weeks ago. When I arrived in town, people on TV and online were still talking about a failure to “deliver” Brexit being met with riots – and as the Tories’ bunfight spluttered on, senior Conservatives banged on about ending free movement “once and for all”, and exiting the EU even without a deal, to give the public what it wants. I have met people who would like that kind of talk, but there are fewer of them than Boris Johnson, the home secretary, Priti Patel, and their colleagues might imagine – and in Milton Keynes, they seemed to be in an even smaller minority than usual.

This is not to suggest that the extremities of Brexit politics are not present. Reflecting the national picture, local police have reported a post-referendum surge in hate crime, and partly put this down to “Brexit banter”. In the recent European elections, the Brexit party topped the local poll. But, given low turnout, its support among the total electorate was about 10% – and, in keeping with this picture, a lot of the people I met talked in strikingly ambivalent, qualified terms about our exit from Europe, and the state of the country. One leave voter taking five from his work as an electrician told me that after three years of political inertia and mounting signs of economic danger, he would now settle for whatever benefited the economy and country; another Brexit supporter, who spends his waking hours caring for his disabled son, said he was now so fed up that he was thinking about voting for the Lib Dems.

In the south of town, I met Donna Fuller, a brilliant, energised community council leader and grassroots social activist who avoids party politics. “Brexit’s going on, and there’s all sorts of arguments around it,” she said. “But actually, that means nothing to the people round here. It’s almost above them … happening to them, not with them.” Implicit in what she said was something still too little understood. People were asked to choose one of two options in 2016, and did as they were told – but that does not mean that the leave voters among them are necessarily minded to follow the prime minister as he rides his knackered stallion to Brussels any more than everyone who voted remain is now desperate for another referendum.

Facebook Twitter Pinterest ‘Amid new-build houses and cutting-edge workplaces, a lot of people are waiting for something different.’ Milton Keynes housing being built in 1975. Photograph: Simmons Aerofilms/Getty Images

Ten days ago the New York Times ran a news feature headlined “The America that isn’t polarized”. It quoted the academic Sam Abrams, a visiting scholar at the American Enterprise Institute. “We are just not seeing the polarisation among the masses that people imagine,” he said. “People who watch MSNBC and Fox are a loud but small minority.” Another voice in that piece acknowledged that the majority of people in the US do not participate in their country’s culture war, but said that political events are now “driven by small numbers that can shame and intimidate large numbers” because “social media has changed the dynamic”. This rings true, and surely applies just as much to Britain as the US. But it also shows that we are in danger of making the classic postmodern mistake of taking spectacle and skewed representation for reality. For most of my time in Milton Keynes, I ignored the moronic inferno of Twitter, and barely watched TV. The result, not entirely surprisingly, was a much more authentic sense of how politics plays out in the world, and a reminder that many voters are alienated by all the din surrounding Brexit, and feel that politics has turned far too shrill and self-righteous.

Lewis Hamilton misspoke on Stevenage’s ‘slums’. In fact, this town can teach us a lot | Gary Younge Read more

I met lifelong Tories who said that Johnson’s arrival at the top, his nasty rhetoric and his do-or-die vision of leaving the EU meant they would not vote Conservative for the foreseeable future. There were also several encounters with past supporters of Labour who had a damning view of Jeremy Corbyn, and seemed to think that he symbolised something comparable to the Tories’ journey to the extremes. I actually don’t think this rules out the kind of radicalism Labour now represents: Milton Keynes is full of the kind of basic social dysfunction, from low wages to a lack of housing, that now runs to almost every part of the country and cries out for answers, albeit framed more winningly than Labour is currently managing. The fact that the party’s local vote shot up two years ago seems to give it grounds for hope. But mistrust of the Labour leader illustrates another element of political estrangement, and how a discourse of bitter partisanship, online shouting and endless faction-fighting – on both right and left – is leaving a lot of people cold.

The fact that this is happening just as Brexit reaches its most critical phase underlines perhaps the biggest irony of our time. Immediately after the 2016 referendum, some of us had a vague idea that the shock of the result might reconnect a swath of the public with politics. But as it has turned out, the absolutist, angry, often nostalgic turns our public debate has taken ever since has only made the gap even wider. Amid new-build houses and cutting-edge workplaces, a lot of people are waiting for something different. Mischievously, one could draw on the vocabulary of old-school conservatism and call them the silent majority. Whether that term fits or not, there are more of them than we think, and they will sooner or later make their voices heard.

• John Harris is a Guardian columnist