The Labour party disintegrates before our eyes. There are obvious reasons: five years of superficial party unity and no real intellectual or organisational advance always meant it would crumble fast after another big defeat.

But something deeper is fuelling the decline: the absence of a part of Labour that combined ideology and pragmatism, that was more democratic, pluralist and green than the rest of the party, and which mediated the hostile relationship between the right of the party and the hard left. It was known as the “soft left” – as against the more sectarian, controlling “hard left” – and its steady drift away from Labour into new areas has left the party light on ideas and unable to balance its two extremes.

As the noise around the leadership election reveals, all we are left with is two extreme wings calling each other “Trots” or “Tories”. The right sees the insurgent Jeremy Corbyn as a hopeless loser, while the left sees Liz Kendall as a value-free zone. By contrast, the soft left could always see the need to win but to do so for a purpose. Without this bridge, the party has become a war zone, where power and purpose can’t be reconciled.

The soft left wanted a dialogue and a future that would be negotiated, not imposed

Some history. The soft left emerged in 1981 after Tony Benn narrowly lost the deputy leadership. It lived, in particular, through the Tribune group of MPs, the Labour coordinating committee, an early network of thinkers, activists and politicians, and from the mid-1980s enjoyed the intellectual sustenance of the journal Marxism Today. Its thinking and organising ability was a critical part of Neil Kinnock’s leadership.

But then things changed. The 1992 defeat collapsed the space for innovative thought and fed an appetite to win at any cost. The eventual party dominance of Gordon Brown and Tony Blair crowded out the space for anything other than a small clique at the top.

New Labour used what remained of the soft left’s politics but soon morphed into the politics of privatisation, and ill-advised military adventure, scorching the political ground around it. Key soft left figures such as Bryan Gould, Peter Hain, Robin Cook and Clare Short were all marginalised. By the turn of the century New Labour squeezed the party so it had no room to breathe.

For people who believed in the soft left vision of pluralism and openness, an enduring Labour culture of command and control proved too tough to crack.

The soft left wanted a dialogue and a future that would be negotiated, not imposed. But the hard men of the left and right could always outmanoeuvre them, and they did. At first glance, Ed Miliband looked like he represented a version of the soft left, but he was always more Brownite than he would admit. For him and his allies, politics was about pulling levers, not mobilising social movements; top-down energy price caps, not bottom-up community renewable energy schemes. Even more space was closed down.

The last intellectual vestige of the soft left is the influential MP and thinker Jon Cruddas. He alone has tried to bridge left and right. But it’s perhaps an impossible task for even the best mind Labour has to offer, as the party shrinks into a small world of blame and betrayal. Compass, the organisation I chair, which embodies the sprit of the soft left, opened its doors in 2011 to greens, nationalists and social liberals – frustrated by being tied to a party that didn’t want to learn from others or share power.

Labour must learn to speak human, whatever its policies | Jonathan Freedland Read more

But the soft left didn’t just disappear. Its people and ideas simply went elsewhere. Whether they self-identify as such or not, the soft left can be found in the work of Hilary Cottam at the public service innovation outfit Participle, Matthew Taylor at the home of modern enlightenment the RSA, Sue Goss at the Office of Public Management, Nick Pearce at the IPPR, Indra Adnan of the Soft Power Network and Robin McAlpine at Scotland’s thinktank Common Weal. It lives through journalists such as John Harris, Zoe Williams, Paul Mason and Will Hutton, in the pages of the New Statesman, on the openDemocracy site and in Soundings, the house magazine of the thinking left. It is being regenerated by academics such as Mariana Mazzucato, Hilary Wainwright, Jeremy Gilbert, Roberto Unger and Frederic Laloux. It still has remnants in Labour but is probably more strongly represented in the SNP, the Greens and even the Liberal Democrats.

The soft left’s challenge, now as then, is to help reconcile Britain to the politics of equality, innovation and modernity. As the new times of a networked society and the sharing economy kick in, it is the politics of the soft left that can help the country navigate this challenging and potentially exciting terrain. Whether Labour wants to be part of this is a decision only it can make. But Britain urgently needs a political vehicle that embraces democracy, equality, pluralism and sustainability in equal measure.