On the really painful days in politics, most commentary isn’t worth the name. It’s not analysis, it’s score-settling; party political broadcasts for the I-told-you-so brigade, rushed out by people who won’t admit to ever getting a single thing wrong themselves.

So it will go this weekend. You’ll read that Labour’s wipeout was only down to Brexit, by those who won’t admit a flaw in Jeremy Corbyn and his noodling mess of a campaign. Or that it was all the Labour leader’s fault, said by remainers who have seen the position they urged on him blown to bits. It was both, of course. Ask voters in those seats that have just gone blue for the first time since the 1930s, or the Labour would-bes who tried canvassing them. If you want to play election Cluedo, then Corbyn and Brexit go together like Colonel Mustard with the candlestick. But when a party descends into civil war, the factions at each others’ throats rarely bother looking up at the rest of the country.

To clear the decks, here’s my confession. I never foresaw the scale of this wipeout – and what it spells for our already failing economy, fractured society and battered democracy frankly scares me. Yet the reporting I’ve done – both in this election and before – made me almost sure Labour was going to lose, and in precisely those areas that are all over the front pages. What were called its heartlands, at least until Thursday night. The Bolsovers, the Bishop Aucklands. The un-metropolitan, unfashionable, never-kissed-a-Tory land that would, as the old saw goes, elect a donkey if it wore a red rosette.

Working-class voters desert Labour as 'red wall' crumbles Read more

And I can say with certainty that this week’s meltdown is the culmination of trends that stretch back decades. They were Corbyn’s poisoned inheritance, not his creation – but any leader who wants to win back those seats will have to deal with them better than he managed.

For decades, their party took much of the north, the Midlands and Wales as its birthright. It was the “red wall” that would repel invading Tory forces. As one Labour county councillor in Derbyshire, the region that lost Dennis Skinner as an MP on Thursday night, told me: “They barely bothered to campaign.” While the party bigwigs threw their weight about, the mines and the manufacturers, the steel and the shipbuilding were snuffed out. With them went the culture of Labourism: the bolshy union stewards, the self-organised societies, most of the local newspapers. Practically any institution that might incubate a working-class provincial political identity was bulldozed.

In North East Derbyshire last month, I saw up close what was left: warehouses and care work. Bullying bosses, zero-hours contracts, poverty pay and social security top-ups. Smartphones to tell you whether you have a shift that morning, and Facebook to give you the news, or some dishonest fragment of it. Across the UK, mines were turned into museums, factories swapped for call centres, meaningful local government replaced by development quangos.

Play Video 18:00 Anywhere but Westminster: how Labour lost, and the hope that endures – video

And what was Labour’s response? Tony Blair and Gordon Brown pretended some new skills-based economy lay around the corner and parachuted their own chosen people into these safe seats. Thinktankers, union HQ bureaucrats, ex-student politicians: all found careers and weekend homes for themselves.

David Miliband swung from the Institute for Public Policy Research to Blair’s office to MP for South Shields – which, as a longstanding Labour parliamentarian told me yesterday morning, “he couldn’t even find on a map”. Last week Miliband claimed on Twitter: “The biggest Labour challenge is not the angst of the middle class … it is the disbelief of the working class.” This tribune of the Tyneside proletariat now works 3,000 miles away at a New York-based charity that in 2017 reportedly paid him £680,000.

What did such smooth-cheeked careerists offer their constituents? Head pats about the “white working class” and their “legitimate concerns”. Never mind that the working class might also be brown.

Even as the working class were marginalised politically and destroyed economically, New Labour patronised them into apathy. As the Oxford political scientists James Tilley and Geoffrey Evans argue, the “decline of class-based voting was driven by Labour’s shift to the political centre ground”. Meanwhile, the big gap in the electoral market that opened up was for a party offering a welfare state with reactionary social policy. That was Nigel Farage; now it’s Boris Johnson. What won on Thursday night wasn’t Conservatism: it was Faragism. The Tories’ key personnel come from Vote Leave and, just as in that campaign, are happy to play with racism. Some in Labour might well think they can win back seats by beating up on immigrants and tacking to the right: Johnson’s party can do both with far more gusto.

Corbynism began with promises of democracy, but ended up as bunkerised as all other Labour leaderships. What started as anti-austerity movement is now a melange of ideas, most of which look and sound utterly absurd on a doorstep on a rainy morning.

In the era of taking back control, Corbyn offered yet more direction from Westminster, with utilities run from the centre and hundreds of billions disbursed from remote state institutions. Many of these ideas are interesting, but few of them were properly worked through and none patiently argued for.

Giving workers stakes in firms, a green new deal, free broadband: each one came well-intentioned but bedecked in question marks. Any radicalism that fails to ask the really thorny questions isn’t radical at all. In Britain in 2019, those include: against rampant inequality and climate change, what’s the economy for? What do the public actually want from politics and economics?

In the 2017 election I wrote that a party that grew out of social institutions needed to turn itself into a social institution in precisely those areas it historically took for granted. That remains the key task: providing advice to those whose benefits are being slashed, legal support to tenants under the cosh from their landlords, haggling with the utilities to provide cheaper and better deals. Add to that: teaching political and economic literacy to voters, not just activists, and consulting constituents on what issues Labour should be battling on.

None of this is as easy as getting the woman with the great backstory to run No 11, or some GCSE marketing talk about finding new “narratives”. It’s hard graft, and it won’t make good copy. But Labour has no God-given right to expect votes, let alone to govern. It needs to renew its contract with its base. The big question is whether it wants to.

• Aditya Chakrabortty is a Guardian columnist