For a sense of the division now tearing through the Labour party, consider a moment that occurred during the Fabian Society conference back in January 2010. The day’s proceedings finished with a Dragons’ Den-style competition for a big idea for the next election manifesto. A pitch for a Green New Deal to provide a Keynesian stimulus, create good jobs, and decarbonise the economy was greeted enthusiastically by delegates but rejected by Gordon Brown’s pollster, Deborah Mattinson, who said that while climate change was “the biggest issue facing humanity” this was not an idea she could sell to voters.

Is this rock star Corbyn's comeback gig – or his farewell tour? | John Crace Read more

There, six years ago, was the essence of Labour’s current civil war: on one side a grassroots bursting with ideas, determined to tackle the most urgent issues; on the other a party establishment so deferential to “political reality” that the survival of human civilisation has to take a back seat. This is the real struggle taking place in the party now: not one between “Blairites” and “Corbynistas”, but between conservatives and progressives.

Unlike those of the Attlee or Thatcher governments, New Labour’s agenda was never a transformative one. Tony Blair saw his task as “to build on some of the things [Thatcher] had done rather than reverse them”. The new economic framework imposed in the 1980s, which dramatically strengthened big business at the expense of working people, was essentially retained after 1997. Thatcherite ideology, with some rougher edges smoothed off, became the neoliberal consensus – more commonly known simply as “the centre ground”.

This was not merely a case of Labour deferring to public opinion, which it was happy to defy over Iraq and the continued corporate ownership of public utilities. New Labour was primarily about deference to the established order, whether or not public opinion could be used as an alibi. At its worst, this led to a kiss-up-kick-down politics, where ministers preferred taking a hard line on asylum and immigration to standing up to the xenophobia of the gutter press, and where the harsh regime imposed on disabled people claiming social security contrasted sharply with the light-touch approach to financial regulation.

The reason Labour’s mass membership now seeks a clear and decisive break with the Blair-Brown model is no great mystery. By the end of Gordon Brown’s premiership, it was clear that the neoliberal turn had been an abject failure. As Professors Kate Pickett and Richard Wilkinson documented in their book seminal 2009 book, The Spirit Level,, Britain was now one of the most unequal countries in the developed world, and as a result also languished near the bottom across a range of key measures of social health and wellbeing. Moreover, those countries at the other end of the league table were the ones that had been most successful in resisting neoliberal trends and retaining the structures of postwar social democracy. Demonstrably, New Labour had taken the wrong path.

Corbyn represents a challenge to a status quo that a broad swath of left-progressive opinion now considers intolerable

The 2008 crash and subsequent period of austerity turned the need to break out of the neoliberal trap into an urgent imperative. With Thatcher’s policy framework and new “common sense” still substantively intact, New Labour’s few progressive achievements proved easy for David Cameron and George Osborne to unravel. Meanwhile, the steady resurgence of racism and xenophobia that began during Labour’s rightward shift on immigration was mixing with the country’s deep economic problems, forming a toxic brew that would lead to Brexit and the outbreak of hate crime that came with it.

Jeremy Corbyn’s support unites around clear basic principles: the need to break decisively with neoliberalism, in favour of a new egalitarian economic model, and to defend migrants, minority-ethnic people and those on social security from the rising tide of bigotry and the effects of spending cuts.

Conspiracy theories about mass Trotskyist entryism, while (unintentionally) entertaining, are of little help in explaining the huge and enduring support for Corbyn among Labour’s ballooning membership. The real reasons are simple enough – Corbyn represents a head-on challenge to a status quo that a broad swath of left-progressive opinion now considers intolerable. No analytical genius is required to identify his limitations as a leader. The problem is the lack of any credible candidate to carry forward that challenge in his place.

That alternative is unlikely to be Owen Smith, a former lobbyist for the pharmaceutical industry who once claimed that Nye Bevan would have supported Blair’s marketisation of the NHS, and likened Labour’s involvement in the invasion of Iraq to socialist volunteers in the 1930s going to fight in the Spanish civil war.

More recently, Smith supported Osborne’s devastating benefit cap because of its popularity with voters, and abstained on a welfare reform bill that was expected to negatively affect 330,000 of the country’s poorest children. Is this the man to drive through root-and-branch reform of British capitalism in the teeth of strong opposition from elite vested interests, and to challenge majority views on issues including welfare and immigration? Somehow, it seems unlikely.

In politics we tend to assume that the dividing line between the status quo and progressive change runs between the two main parties. In Britain today, however, the line runs right through the supposed party of the left. The implications of this are clearly challenging, but they cannot simply be wished away.

• This article was amended on 29 July 2016. An earlier version used the word “effect” where “affect” was meant.