The chances of the Labour leadership contest flaring into any kind of life currently seem remote, especially when it comes to meaningful debate about the party’s crisis. There are occasional flashes of candour, such as Lisa Nandy’s insistence that “if we do not change course, we will die, and we will deserve to”. But the contest’s default position is embodied by its two frontrunners. Keir Starmer and Rebecca Long-Bailey have so far displayed one common trait: trying to convey a sense of purpose while saying nothing much at all.

To any outsider, the scale of the party’s predicament is surely clear. Yes, its estrangement from its old heartlands goes back decades, but Jeremy Corbyn’s leadership deepened and accelerated it. Just look at that list of places Labour no longer represents, from all three seats in Stoke-on-Trent, through an array of former mining areas, to places such as Redcar, Scunthorpe and Grimsby. In Scotland, the party is all but extinct.

The question of what now constitutes the working class, and the extent to which Labour speaks for it, is complicated by elements of those parts of society that have remained relatively loyal, the best example being ethnic minority voters. But note that the party lost badly even in such urban, diverse places as Wolverhampton, and statistics suggest a long-term political shift has definitely turned critical. According to YouGov, among skilled manual occupations Labour received only 31% of votes, compared with the Tories’ 49%. Among semi-skilled and unskilled manual workers, the percentages were 34% to 47%.

Some commentators seem reluctant to acknowledge that Labour has huge problems. I have read that the defeat was not as bad as the party’s beating in 1983, and there has been Kremlinology about the relationship between the party’s hard and soft left. This is surely little more than displacement activity. In search of enlightenment, one option is to return to things written many years ago, whose diagnoses of Labour’s essential problems remain uncomfortably valid. One is an anthology of writing from the left published in 1989. New Times, an offshoot of the magazine Marxism Today, was produced by a small group of people clustered around what remained of the British Communist party, many of whom were only really Marxist in the sense that they understood that political shifts usually have deep roots.

As they saw it, Thatcherism was successfully responding to tectonic changes in the economy and society, while the left was stranded in a “Fordist” politics built around the factory, and a monolithic working-class identity that had long since faded. One central problem was clear: “The labour movement is deeply defensive of its main historic achievement – the corporatist, Keynesian welfare state … It wants to occupy the state, when it needs to be transformed.”

Labour’s 2019 manifesto represented this unchanged impulse in spades: if it had a central idea, it was to deliver to a monolithic body of people it called “the many”, massively expanding the reach of government via spending programmes and nationalisation, while leaving the basic structures of the state untouched. Some of the policies were popular in isolation, but the public recoiled from the whole package for many reasons – some bound up with trepidation about the cost, others that cut straight to the left’s failure to understand that 1945 was a long time ago.

Play Video 2:29 Jess Phillips quits Labour leadership race saying she 'cannot unite the party' – video

The central state remains the only means of taxing the rich, regulating big business and setting frameworks for climate action. But try selling the idea of big government to people well acquainted with the regime at their local jobcentre, those who have tried to get an unresponsive public sector to help their disabled child, or the multitudes whose experience of the state often boils down to endless form-filling and half-hour waits on phone lines.

Popular revulsion towards bureaucracy has always blurred into a perception that the left’s prime movers tend to be distant, arrogant and pious. Back in 1937, George Orwell pointed out that “to many people calling themselves socialists, revolution does not mean a movement of the masses with which they hope to associate themselves; it means a set of reforms which ‘we’, the clever ones, are going to impose on ‘them’, the lower orders”.

There is a modern version of this problem, bound up with a combination of old-fashioned statism, Labour’s increasingly middle-class makeup, and the way the left’s focus on the politics of attitudes and behaviour sometimes teeters into shrill intolerance, not least online. By comparison, Conservatism’s eternal promise is that its supporters will be left alone. Millions of people will always vote for that – not just because it represents a quieter life, but because it chimes with the internet age: the fact that people now have a voice, and don’t like being told what to do, or who to be.

This is not the only sense in which the left is out of step. The leaders of the big trade unions are still male, old and white, highlighting the failure to renew and rebuild. (So, it’s likely, will be the next party leader).Meanwhile, although in the Corbyn period Labour recruited thousands of energised, idealistic people who will be central to its future, a key problem remains. In much of the country, millions of people to whom the party and its movement ought to speak with confidence – many young, women, or both – know little of the labour movement and how it might help them.

In turn, a lot of people on the left seem to know nothing of them, as evidenced by that dread expression “left behind”, or the current belief that Labour’s lost heartlands are synonymous with angry men, nostalgia and “social conservatism”. They are not really like that at all: large swaths of Labour’s lost territory are smattered with trailblazing social projects, often run by women, with future-facing and often cruel local economies built around logistics and distribution.

As old narratives of class and heavy industry have faded away in these places, some people have hung on to identities based on place and nation. Yet these are much more complicated than the caricature of stupidity and racism: witness the fact that a third of ethnic minority voters backed Brexit.

Play Video 3:45 What do the Labour leadership candidates really stand for? – video explainer

The same problems afflict progressive parties across Europe, and even starting to address them will be a long and complex business. Trade unionism will have to be completely reinvented: people are not just workers, but parents, carers, volunteers and consumers. Socialists will have to become determinedly localist, and Labour will have to put the people who run towns, cities, counties and boroughs at the forefront of its campaigning, learning from what they do.

'If Labour doesn't talk to the voters it lost, I fear for the future' Read more

Contrary to the laughable top-downism seen recently in Momentum offering its members a yes/no verdict on candidates its high-ups had already picked, the left will have to open itself up to the new world of citizens’ assemblies and open primaries. It also needs its own national story that runs deeper than slogans about “progressive patriotism”, and a cleverer means of fighting the supposed culture war than repeatedly telling half the population they are bigots.

All this would begin a shift in consciousness that is at least 40 years overdue. But first, if the precious values of equality and solidarity are to endure, some people on the left will have to do something that has never come easy: stare into their own crisis, and acknowledge that their party and movement are still stuck in the 20th century while their adversaries speed into the future.

• John Harris is a Guardian columnist