The Labour party has always been a broad church made up of many political traditions: from trade unionists, to free marketeers, to radical socialists. With each leader the makeup of the party has changed composition, bringing in new members and shifting old alliances. The Corbyn years saw a significant sea change in Labour’s internal politics, and the current leadership election is revealing new contours to Labour’s internal topology. Most important, we’re seeing the emergence of a distinctive, radical-left current that is democratic, green and internationalist in its socialist aspirations.

Until 2015, there were four main factional tendencies in the Labour party: the “old right”, the “hard left”, the “soft left” and the Blairites. The old right – rooted in local government and union bureaucracies – has campaigned against radical socialism since the 1940s. The political crises of the 1980s saw the Labour left divide between the hard left of Tony Benn and the soft left led by Neil Kinnock (and, later, Ed Miliband). The soft left wanted to update socialism for a post-industrial age, to expel Trotskyist factions from the party, and to make whatever accommodations it took to win elections. The hard left remained committed to the radical policy agenda developed in the 1970s, despite waning support for traditional socialism among the electorate. The Blairites, advocating free markets and globalisation, emerged as a distinctive section of the party elite in the 1990s, but never had an enthusiastic base among members; they always relied on support from the old right and the soft left to carry out their agenda.

Under Jeremy Corbyn’s leadership, the lines separating these factions blurred and shifted. Several of his senior staff and union supporters came from, or maintained close links with, the hardest sections of the hard left, such as the tiny Communist party of Britain. But much of Corbyn’s activist base – and allied projects such as The World Transformed, Novara and Red Pepper – have occupied a different political space: green, libertarian, pluralistic and democratic. This “radical left” has always been ideologically closer to the most progressive strands of the soft left (such as Compass), which in turn backed Corbyn throughout his leadership, even while more conservative sections of the soft left peeled away to back Owen Smith in 2016. The organisation set up in 2015 to defend Corbyn’s leadership – Momentum – has always contained elements from all of these tendencies.

Labour leadership: Keir Starmer on course to win in first round – poll Read more

In fact, it’s clear now that the old division between the soft and hard left has been giving way for some time to a new distribution of positions. While all three tendencies pulled together under Corbyn, the differences are becoming increasingly visible between the soft left, radical left and a new “orthodox left” as the Labour leadership election progresses.

Keir Starmer’s success in the leadership race so far has been in winning the support of the traditional soft left, which still constitutes a large proportion of party members. Despite the failures of both Kinnock and Miliband, their default assumption remains that progressive government can be achieved by selling moderate social democracy to the electorate, led by a guy in a smart suit. At the same time, the old right and the Blairites have all accepted that, for now, they can only gain influence by backing a soft-left contender such as Starmer. Nandy has pitched herself at times to the soft left, at times to the old right; and so has won significant support from neither.

Linked to the leadership of unions such as Unite and the RMT, the orthodox left back Long-Bailey as Corbyn’s anointed successor. Under any other leader, their influence will drastically decline. The orthodox left still basically wants to implement the Communist party’s 1951 plan, The British Road to Socialism, with its vision of socialism being implemented in one country by a strong, centralised national government. They lean heavily towards a pro-Brexit position, while tending to interpret support for Brexit among working-class voters as incipient class consciousness rather than tabloid-inspired xenophobia.

The radical left is still a very new, fragile and inexperienced tendency that has a long way to go before emerging as a mature political formation. It brings together the more libertarian strands of the hard left, the more radical strands of the soft left, and a new generation of activists from outside the traditions of the Labour party. Clive Lewis’s short-lived leadership campaign was the most obvious expression to date of the desire of some Labour members to express a distinctive left position that is more green, more internationalist, more democratic and less tied to traditional Labourism than that of the orthodox left. Long-Bailey shares much common ground with this tradition, having championed Labour’s plans for a green industrial revolution; she has since got further by backing open selections. But it is mainly out of sheer loyalty to her mentor, John McDonnell, that most of the radical left have supported her.

This is an uneasy alliance. Privately, many on the radical left agree with former MP Alan Simpson that the dogmatic and authoritarian tendencies of the orthodox left smothered the creative and democratic potential of Corbynism, contributing to its eventual downfall. For example, the leadership of Unite and other unions have opposed open selections of parliamentary candidates, preferring to bolster their own direct influence over candidate selections, despite democratisation of the party being Corbyn’s first promise to his followers, and Momentum’s major demand. On the other hand, there seems little chance of the radical left becoming anything more than a fringe movement unless it maintains its effective coalition with the orthodox left: the latter being far more established and empowered at the level of effective institutions, from Unite to the Morning Star.

But for that coalition to be maintained, the radical left may need to find its own distinctive organisational voice. Whether that will take the form of some new formal network or actual membership organisation, or whether the radical left will remain a loose affiliation of activists and alternative media, remains to be seen. But even in the very likely event of a Starmer victory, it is hard to see a future in which Labour confronts the real challenges of our time – the climate emergency and the breakdown of representative democracy – without the energy of the radical left driving it forward.

This radical left is not a brand new phenomenon. In many ways it inherits the legacy of the New Left of the 1960s, and traditions of radical democratic socialism going back to the 19th century. But arguably its egalitarian, libertarian worldview is now shared by more people – especially younger people – than at any previous time, both in the UK and the US. One of the most distinctive features of the movement to elect Bernie Sanders has been its embrace of just this participatory, horizontal, green and libertarian politics (just compare his position on drugs policy to those of the Labour leadership contenders). This is the shape of 21st-century socialism.

Labour suffered a terrible electoral defeat in 2019, but it still won more votes than in 2015, 2010 or 2005 under Tony Blair, and the recovery of its activist culture and support among young people has been remarkable. These gains were all won by the energy and imagination of the radical left, and they will have to be maintained and built on if Labour is ever to form a government again. Any future leadership will have to include elements and ideas that can represent this tendency and give it room to grow.

• Jeremy Gilbert is professor of cultural and political theory at the University of East London and the author of Twenty-First Century Socialism

• This article was amended on 2 March 2020. An earlier version was incorrect to say that in 2019 Labour had won its most votes since 2001. Although it won more votes in 2019 than in 2015, 2010 or 2005, it did not win as many as it had in 2017.