After a long tumble into disgrace and confusion that dates back well over a year, the story playing out at the top of the Labour party increasingly seems to be so dreadful that it defies belief. Jeremy Corbyn is doing a very good impression of someone who would rather be anywhere else. The party’s supposed leap forward on Brexit policy seems to have resulted in a stance not quite as bamboozling as the one it replaced, but it is still surrounded by unanswered questions; the leadership’s dearth of collective energy as the agenda of a Boris Johnson government takes shape is miserable to behold. Obviously, the people at the top have other things on their minds. Woven into everything is the ever-widening story about antisemitism, which now includes signifiers for almost every aspect of the party’s malaise – from the presence in the party of hateful attitudes towards Jewish people, through allegations of the awful treatment of young party staffers by powerful people at the top, to the sense of any sensitivity and seriousness now being drowned out by the familiar sound of belligerence and faction-fighting.

All too often, anti-imperialists seem to keep the company of very rum people indeed

Faced with all this, there is a continuing view, among some Labour MPs and party grandees, of Corbynism as something entirely irrational and aberrant – a shambling monster that took over the party thanks to a series of accidents, and has been sustained ever since via the mysterious magic of the internet. This view is often part of the kind of political restorationism lately manifested in nostalgia for the Tony Blair years, and the idea that if only we could go back to an unspecified time between 1997 and 2006, everything would be OK. The broadcast media tends to privilege this view of things, along with Corbyn-till-I-die posturing: the result, as evidenced by what happened on TV and radio last week, is the seemingly daily spectacle of backers of the leader duking it out with older, often jaded people who treat them and their politics as something almost beneath contempt.

At which point, a reminder: four years ago, to his own evident shock, Corbyn raced ahead in the Labour leadership contest that followed the election defeat of 2015 for a set of clear and undeniable reasons. For all that Andy Burnham and Yvette Cooper have since risen to be, respectively, the mayor of Greater Manchester and one of the most impressive Commons critics of the Tories, the vapid offerings of his adversaries in the contest (the third, you may recall, was the Leicester MP Liz Kendall) was plain to see. In the moment, it was very exciting to see so many people gathering, both in the real world and online, to voice their opposition to austerity, the cruelties of the modern benefits system, homelessness, and all the other moral outrages of 21st-century Britain.

Around this time, a lot of people became Labour party members – and are still involved – for the most straightforward of reasons. When I went to an early Momentum gathering in Kent, I met a woman who told me she had never considered of any kind of activism before, that she lived “constantly on the poverty line”, Labour was at last speaking to her predicament, and that Corbyn offered an alternative to “how people think a politician should talk, and look”. Thousands of people had their own version of this story, and as evidenced by the yearly Labour-aligned festival of ideas titled The World Transformed, many of them soon set about creating a largely open and thoughtful – and passionately green - movement that brimmed with energy and ideas.

But thanks to an accident of history, these people’s politics came to sit alongside and often blur into a cliquey, closed-off strand whose roots go back into the mists of the British left’s past. Alongside a belief in top-down power structures, among its key features are hostility towards the EU, and an affinity with the old Soviet Union that is now manifested in sympathy with Vladimir Putin. These things go with the grain of a supposed anti-imperialism that doesn’t only criticise US foreign policy and Israel’s treatment of the Palestinians, but sees those things as unsurpassable evils. All too often, its adherents seem to apply entirely different moral standards to events depending on who is held to be responsible (witness responses to foreign intervention by Russia and the US), and keep the company of very rum people indeed: the tendency of some anti-imperialists to associate with or endorse antisemites is part of the reason why Labour’s current problems started in the first place.

At the grassroots, this stuff tends to be voiced by the kind of Labour member who often believes in a tangle of conspiracy theory, interprets most challenges to Corbyn as evidence of plots and hidden schemes, and sees alarm on the left about Brexit as a bourgeois indulgence. Higher up the party hierarchy, in a much more controlled and outwardly sensible form, a related set of views is there in Corbyn’s own politics, and those of such far-left veterans as his chief of strategy, Seumas Milne, and the leader’s close adviser Andrew Murray, both of whom who cut their teeth in the culture around doctrinaire British communism, and the political forces who eventually formed the deeply inconsistent and compromised Stop the War coalition.

The separation between these factions and the new Labour party’s more modern elements is not always clear: out of loyalty to the Corbyn project, the latter often defend and support the former. But nonetheless, in the awkward fit between them there lie issues that the left’s polarised discourse has so far overlooked. Why, if you want British society to be radically changed, the climate emergency tackled via a Green New Deal and the basic notions of democracy and empowerment rolled out into the economy as well as the political system, does that agenda have to be coupled with fringe ideas that only speak to a tiny minority of people, and have played a key role in Labour’s current mess? Put another way, why does 21st-century socialism have to be bundled up with all this weird stuff?

Even if it has yet to cohere into a convincing narrative, there is an abundance of good ideas in and around Labour and the left, from John McDonnell’s plans to break up the Treasury and move a lot of its work to the north of England, through ideas about punishing companies that fail to pull their weight on climate change, and on to mounting a push to make co-operatives a central part of the economy. By an accident of timing, the night last week’s Panorama about Labour and antisemitism aired, I was in the basement of the HQ of the trade union Unite, speaking about a report jointly authored by the Runnymede Trust and the Corbyn-aligned thinktank Class, about the modern working-class experience in London, and a set of policy proposals that ranged from a set of universal citizen entitlements to housing, transport and internet access, to the end of the sell-off of public land. Such organisations as the Institute for Public Policy Research and the New Economics Foundation are developing the central idea of the drastic redistribution of power into exciting, trailblazing work.

This has to be Labour’s future. As Britain edges closer to a post-Brexit reality in which the political right will do its utmost to snuff out whatever residues of collectivism and solidarity we have left, Labour urgently needs to attend to it. Whether it can do so while its collective politics is so messed-up, inconsistent and toxic is the question that is already starting to decide its fate.

• John Harris is a Guardian columnist