When Fenner Brockway, the Labour MP, lifelong anti-imperialist and peace activist, recalled his early involvement in the Independent Labour Party, he wrote, “On Sunday nights a meeting was conducted rather on the lines of the Labour Church Movement – we had a small voluntary orchestra, sang Labour songs and the speeches were mostly Socialist evangelism, emotion in denunciation of injustice, visionary in their anticipation of a new society.”

Fast-forward a century or so, and Brockway could be describing a Jeremy Corbyn leadership rally: the same joyfulness, fervour of conviction and ecstasy of expression, only this time clothed in the self-belief of the Labourist left, rather than Nonconformist millenarianism, and playing to a larger crowd. Corbyn’s campaign reinvented the party political rally, a form of British politicking long since presumed dead. He created a space in which the lost tribes of the British left could reunite, and new followers join the throng. Suddenly a “surge” was under way, a democratic explosion within the mainstream body politic, not safely contained outside it.

Corbyn’s election as leader of the Labour Party is undoubtedly a seismic event. But it does not herald a wider political transformation. For although the left of the Labour Party is not a sect, it is sectarian. It inhabits a world-view, culture and practice of politics that is largely self-referential and enclosed. Save for brief moments of popular experimentalism – such as the two occasions when Ken Livingstone governed London – its reach has been minimal. Corbyn’s policy platform is an unreconstructed Bennite one, defined by nationalisation and reinstatement of the postwar settlement, given a fresh lease of life by revulsion at foreign wars and the social consequences of austerity. While his campaign tapped into discontent with the decrepit state of mainstream Labour politics, it did not give birth to a new social movement, rooted in popular struggle, like those that have sprung up in southern Europe. His improbable leadership of the Labour Party is another symptom of the crisis of social democracy, not the incubator of its future.

That social democracy is in crisis across Europe is indisputable. Few parties of the democratic left now register more than 30 per cent in national elections. In its northern European heartlands, social democracy is either besieged by populist anti-immigrant parties or marginalised by a dominant centre right. Even in Germany, where a recognisably social-democratic culture still exists, the SPD is reduced to junior-party status, topping out at 25 per cent of the electorate. Elsewhere, austerity has either destroyed the mainstream left, as in Greece, or cut it back to its core, as in Spain and France. Only in Italy, where the right has been discredited by years of corruption and abject economic performance, does the centre left have any energy.

Britain’s first-past-the-post system has protected the Labour Party from the full force of these currents, but the pull of their logic is at work here, too: political loyalties have fractured, immigration has split the core working-class vote, and the financial crisis has ushered in a politics of economic security, not reform.

The last time the death notices of social democracy were written in the early 1990s, a wave of Third Way revisionism brought it back to life. Then, social-democratic parties expanded out of their working-class electoral heartlands and public-sector redoubts, forging new coalitions of support. The freshly modernised centre left won power across Europe and in the United States. But the breadth of its appeal was not matched by depth. Over time, centrist voters proved fickle and the core vote started to abstain or desert to the anti-immigrant right. Centre-left parties began to shed votes and lose power. The financial crisis provided the coup de grâce, punishing incumbents and passing the baton of energetic opposition to new parties of the left such as Syriza and Podemos.

Today, it is clear that Third Way modernisation relied on historical circumstances that cannot be repeated now: principally a long wave of growth, in which a build-up of household debt and government transfers maintained living standards, despite rising asset inequality and the sundering of the link between productivity increases and wages. “Globalisation plus good schools” is no longer a plausible formula for winning back working-class voters, and the fiscal headroom for binding the middle classes into an electoral coalition built on investment in public services has shrunk. Nor can the rise of identity politics, whether of the civic nationalist or the anti-immigrant kind, be properly understood, let alone contested, within a political strategy that gives pride of place to individual social mobility. Even the crowning achievement of the New Labour era – the rescue and revitalisation of public services – would now require a very different set of tools from the centralism of the turn-of-the-century delivery state.

***

In the early 1990s, New Labour thinkers looked across the Atlantic for inspiration and renewal. Bill Clinton’s insatiable curiosity for policy ideas rubbed off on Blair and his advisers, but the most important lessons were strategic: how to win back voters in the mainstream of politics and push the right off the centre ground. Today, the transatlantic cable is broken. Latino migration to the US has replenished the Democrats’ vote base and refreshed its politics, while immigration has done the reverse to European social democrats. The White House cannot be won with older white voters, but, in Europe, ageing societies have become more conservative, making it harder for the reformist left to win. At the last general election, Labour won every age group up to those aged 55 and over, but haemorrhaged support among pensioners. The party’s Russell Brand moment never arrived. Inequalities of turnout between young and old, prosperous and poor, are such that it likely never will.

The conservatism of ageing societies, the cultural and political fracturing of the working class, and the structural dysfunctions of debt-laden western economies all pose grave challenges to social-democratic parties. The task is magnified for Labour by the break-up of the political unity of the British state, and the collapse of its support in its Scottish heartlands. Unlike in the 1980s, it cannot fall back on the ballast of a centrist trade union movement and cross-national solidarities of class.

More serious still, its intellectual resources are depleted, left and right. Those who have sought to renew Labour at critical moments in its history have always had to battle against a deep strain of anti-intellectualism in the party. Because it famously owes more to Methodism than Marx, it has never possessed a theoretical tradition. In the 20th century, it borrowed heavily from Liberal giants such as Keynes and Beveridge, and turned to the Fabians and the London School of Economics for technocratic expertise when economic planning and the construction of the welfare state demanded it. But it only ever produced a few big thinkers of its own, such as Tawney, G D H Cole and Crosland, and even their influence on the course of Labour politics was limited. When it last faced the prospect of terminal decline, in the 1980s, it had almost no intellectual resources to fall back on. Instead, it was the Gramscian thinkers grouped around Marxism Today who furnished it with an analysis of Thatcherism and a route map towards re-election.

New Labour’s openness to wider currents of ideas – at least in its early, formative phase – allowed it to draw on fresh thinking from academia, think tanks and elsewhere. But the Labour Party’s intellectual revival in the late 1980s and 1990s owed much to a cadre of soft-left MPs, epitomised by Robin Cook and Gordon Brown, who could act as receptors into the labour movement of the thinking that was taking place outside it. No such cadre exists today. The soft-left tradition was weakened by defection, desertion and (tragically, in Cook’s case) death, and what remained of it in the parliamentary party at the turn of this century had become a Brownite patronage network. Ed Miliband failed to revive it, despite being suited to the task. The reductio ad absurdum of this decline was reached in the desperate political gymnastics of Andy Burnham’s leadership campaign.

Labour’s anti-intellectualism would be less of a problem if the party were well attuned to public sentiment and capable of intuiting the sources of change in British society. But it is not. Like other mainstream political parties, it has become hollowed out, professionalised and state-centred in recent decades. As the class structures that gave birth to Labour politics declined in the second half of the 20th century so, too, did the party’s roots in civil society begin to shrink. Its forms of popular culture, its institutions and its membership base all withered, leaving it with leaders drawn from a professional caste, possessed with all the skills and networks necessary to navigate Westminster and Whitehall, but with not much underneath or around them in the wider society.

This decline has been apparent since the late 1970s – certainly since Eric Hobsbawm wrote his celebrated essay “The Forward March of Labour Halted?” (1978). And yet, despite significantly broadening its electoral appeal in the New Labour era, Labour has not created social and economic bases to replace those lost with the passing of industrial society. It has become caught in what the political scientist Peter Mair diagnosed as the trap facing all centrist parties: the one between responsibility and responsiveness. Parties aiming for elected office seek the patina of responsibility, fiscal and political. They set out credible, carefully crafted programmes for government, mindful of its constraints and compromises. Instead of representing the people to the state, they increasingly represent the state to the people. This leaves the field open for populists, who eschew responsibility in favour of responsiveness, unmediated authenticity and the articulation of an anti-politics. In recent years, only the SNP has sprung this trap, combining broad appeal with seriousness of governing purpose.

***

Corbyn’s surge did not reverse this decline. The number of trade unionists voting in the 2015 leadership election was lower than that in 2010, and even the addition of registered supporters did not push the selectorate back up to where it was in the mid-1990s (he is also now learning that leadership itself can’t be dissolved into networks, and that the task of leading demands considerable skills). Yet Corbyn’s campaign held up a mirror to the Labour Party, showing it how shrunken, uninspiring and detached from society it had become. Over the course of a few months, he mobilised 16,000 volunteers, pulled in thousands of new activists, and showed the Labour high command how to do digital politics. Some of his supporters are day trippers who won’t stick around. But many more are for real, with decent intentions; and they have changed the party irrevocably. Corbyn used Labour’s new internal democracy to open the party up, and in so doing placed the cadaver in full view. There is no going back.

Is social democracy finished, a relic of 20th-century class society, as John Gray and others predicted three decades or so ago? Its twin historic tasks – to tame and humanise capitalism, while harnessing its dynamism – remain as valid and pressing as ever. But in this post-crash era, it needs to equip itself with new economic reform agendas. Croslandite and Third Way revisionism were both creatures of eras of economic moderation, and shared a conviction that capitalism had overcome its contradictions. The great financial crisis of 2007-2008 destroyed those assumptions, and threw into sharp relief the challenge of stabilising highly financialised economies while reducing the inequalities and imbalances to which they are subject. Despite his political failure, Ed Miliband was undoubtedly right to see this as the most important challenge facing contemporary social democrats. Without being able to offer more widely shared prosperity, generated from within market economies, and not just by redistribution, social democracy is purposeless.

The intellectual resources for this renewal are readily to hand, in both new Keynesian and heterodox economic thinking, as well as a welter of empirical analyses of central policy challenges, such as productivity and wage growth, household indebtedness, and so on. Indeed, far more original new economic thinking has come from the centre left since the financial crisis than from the right of politics, where think tanks and commentators rehash comfortable Thatcherite nostrums. Politically, however, the story is reversed. Labour’s economic credibility has been shot to pieces since the recession and the party shows no signs of knowing how to restore it. Simply opposing austerity will not do the trick, and arguments about the deficit – let alone quantitative easing – will be otiose by 2020, unless the global economy tips back into recession (and relying on that eventuality would be unwise, if not reckless).

More fundamental still, Labour and its sister parties in Europe have yet to work out how to build broad coalitions for economic reform, in the absence of the strong trade unions and organised workers’ movements that they had at their back in the postwar period. The growth of self-employment, the spread of automation, and the decline of public-sector jobs are all making labour itself more disorganised and therefore harder to mobilise politically. Meanwhile, older voters turn a deaf ear to labour-market concerns. If they are on zero-hours contracts, they are likely to be content with them. If not, they are concerned about savings, asset prices and stable inflation. Even in countries with strong trade unions and large manufacturing sectors, there has been a substantial growth in flexible service-sector employment, and a concomitant decline in the political muscle generated in the workplace.

If nothing else, Corbyn’s victory is a dramatic forcing mechanism for the mainstream of the Labour Party to confront these challenges. A generation of Labour MPs and activists grew up in the shadow of Blair and Brown, and now must shoulder the burden of rebuilding the party without the intellectual and political leadership they once took for granted. They are now freed from the narcissistic feuds and rivalries of that era, but this liberty comes with the heavy responsibility of toiling hard to haul the party back. The scale of their defeat is such that cosmetic change will be wholly inadequate. Corbyn’s campaign showed up the profound individual organisational and intellectual weaknesses of the old-right, New Labour and soft-left wings of the party. The soft left vacillated hopelessly and the old right, deprived of the unions and the power of its MPs, had little, if anything to offer. Blairite standard-bearers were blunt and unforgiving in their analysis of Labour’s 2015 election defeat, but they had no answer to the mobilisation taking place in front of their eyes, nor did they have the magic ingredient that had once made them so successful, of what Hobsbawm in 1988 called “having the future in your bones”. They cannot now retread their old path to power.

***

The character of the Labour Party that emerges from this tumult will tell us whether it has a future as a serious political party. Corbyn’s paradox is that he harnessed democratic energy to a familiar statist and dirigiste project. Labour can only hope to renew if it embraces the democracy and ditches the dirigisme: if any part of 20th-century social democracy needs consigning to history, it is the preference for centralist standardisation and bureaucratic public administration. There are strong currents of both liberalism and conservatism in contemporary Britain, but each shares a hostility to remote, dominant power, whether in the state or in the market. Many of the most liberating contemporary social and economic trends, not least the diffusion of digital technologies, point in the direction of individual empowerment and political decentralisation. Labour has been too slow to grasp this.

Importantly, political and economic dynamism in capitalist economies today is increasingly concentrated in our cities, and this is where progressive politics is strongest. Although national elections cannot be won with cosmopolitan voters alone, city leadership is a vital source of energy, and many of Labour’s best politicians are now found in the town halls and civic offices of Britain. These leaders will be a critical building block in Labour’s renewal, whenever it comes. But that will require the party to understand and embrace the devolution of power, rather than tolerate or, worse still, reject it.

Class reductionism on the Corbynite left gives it a tin ear to the claims of territory and patriotic identity, as well as the demands for power currently swelling across the UK, not just in Scotland, but in England, too. Unchallenged, this will place Labour on the wrong side of one of the most important vectors of British politics: the reconfiguration of the UK as a federal (or quasi-federal) entity. The rise of the SNP cannot be accounted for as an expression of anti-austerity

politics, any more than the demands for greater recognition of English identity can be reduced to anti-immigrant sentiment. Both are expressions of deeper underlying historical changes in the Union, as well as the importance of culture and identity in politics. Without sensitivity to these claims, and an awareness of their democratic potential, Labour will become marginal or irrelevant, when it should be transformative.

There are grounds for optimism on the centre left. Economic reform, meeting the challenges of climate change and ageing, and the promise of digital technologies – all of these hold progressive potential. Social democracy could be just as well placed as any other tradition to capitalise on what the 2020s will bring; it doesn’t need to remain trapped between hollowed-out centrist technocracy and revanchist state socialism. But the depth of the crisis it faces demands deep and sustained rethinking, as well as political reorganisation. The rupture that Corbyn’s election has forced must be a catalyst for that change, or it will never come.

Nick Pearce is the newly appointed Professor of Public Policy at the University of Bath and the outgoing director of the Institute for Public Policy Research. He writes here in a personal capacity.