It is one of the great ironies of our times. While inequality returns as an urgent concern in Western democracies, social democratic parties are in crisis. From France to Austria and the Netherlands to Italy, the mainstream left is losing. The French economist Thomas Piketty sums up the puzzle. His book Capital, charting the rise of inequality, has sold millions of copies. Translating his research into a political choice, Piketty backed Benoît Hamon, the Socialist Party (PS) candidate in the 2017 French presidential elections. Hamon came fifth, the former ruling party lost most of its deputies in the following month’s legislative elections, and the PS put its historic Rue de Solférino headquarters in Paris up for sale in order to pay off its creditors.

Popular concern about the injustices of contemporary capitalism are not doing much for the fortunes of the centre left in Europe. The rise of populist radical-right parties is well documented but the more powerful trend is one of fragmentation across the political spectrum, on the right and the left. Much of this fragmentation is driven by the unravelling of the social democratic left as a credible political force.

In many countries, centre-left governments have presided over a decline in the share of national wealth going to wages, to the benefit of shareholders and landlords. Inequalities in how the wage share is divided up have ballooned as company directors reward themselves with ever greater payouts. The concentration of wealth at the top of society has eroded social norms of solidarity and hollowed out public institutions. This is true even for countries with a strong social democratic culture, such as Norway. Governed by a mixture of centre-left and centre-right governments, with a long centre-left interlude between 2005 and 2013, the famously egalitarian Norwegian social contract is coming apart under the twin pressures of increasing income inequality and booming property prices.

The failure of the centre left to set the political agenda in the wake of the 2008 financial crisis was notable. This period turned out not to be the social democratic moment that the former Labour leader Ed Miliband and many others believed it would be. Instead, it was a period of fiscal retrenchment, or “austerity”, as governments grappled with debt crises.

Faced with such an ineffectual centre left, voters have come to prefer the authenticity of a far-left firebrand, such as Jean-Luc Mélenchon in France, or the curious blend of populism and the promise of more effective government found in the Czech Republic’s Andrej Babiš, Italy’s Luigi Di Maio and France’s Emmanuel Macron.

The picture is not entirely bleak, however. Transformations taking place on the left across Europe suggest both decline and renewal. Change is happening within the existing political parties themselves (such as the Labour Party in Britain) and through the rise of new parties and new citizen movements. And the future of the left in Europe depends on the fate of these different national transformations.

Yet there is also a bigger issue lurking in the background, which is whether the left should embrace or reject closer European integration. The left’s uncertainty over the question of Europe is one of the biggest obstacles to its renewal.

Few parties illustrate better the combination of decline and renewal than the German Social Democrats (SPD). The social conditions in Germany are favourable for a party of the left. The transformation of the German labour market in the 2000s produced a “Hartz IV” generation – named after reformer Peter Hartz – for whom work takes the form of fixed-term or part-time contracts, modest wage growth and a rising sense of financial insecurity. The social and cultural divide between west and east Germany, nearly 30 years after reunification, is still there. It is precisely the gap between the booming aggregate economic performance of the Federal Republic and the everyday reality for many Germans that one would expect the SPD to exploit. After all, capitalism is very much about making promises that it cannot deliver. But instead of capturing the votes of Germans fearful about their economic future, the SPD has lost them to Die Linke on the far left and the Alternative für Deutschland (AfD) on the far right. The steep decline in the party’s vote share – from over 40 per cent in 1998 to just over 20 per cent in 2017 – has been blamed on a loss of identity after years of grand coalition with Angela Merkel’s Christian Democrats.

Martin Schultz, who resigned abruptly as SPD leader in February, had argued that time in opposition would allow the party to rebuild and rediscover its roots. This was a strong argument but Schultz reversed his decision and entered into negotiations with Merkel. In the event, his brand of careless opportunism cost him the leadership of the party and any hope of a job in the new coalition government.

****

At the beginning of March, the SPD’s 460,000 members voted in large numbers to enter a coalition with Angela Merkel. Nevertheless, the SPD debacle demonstrates that senior members of Europe’s left-wing parties are often an obstacle to improving the left’s fortunes.

Having returned to power in the 1990s, after embracing the free market in the name of the new so-called Third Way politics, these figures have become so accustomed to the privileges of office and of power that they have lost sight of their primary purpose of representing citizens. The conflict within the SPD suggests that the party’s grass roots, and its youth wing in particular, has had enough of this sort of self-interested behaviour.

The internal struggle of the SPD over whether to enter into another grand coalition also brought out a deeper generational conflict evident in many advanced democracies in Europe. Wealth is increasingly concentrated among older people, particularly in countries where housing booms have left those in their fifties, sixties and seventies sitting on huge cash piles, to which can be added generous final salary pensions. Some of that wealth is now working its way down through inheritance – the famous “Bank of Mum and Dad” – but much of it is not. The complacency of the German SPD leadership is the complacency of an entire generation: the revolt of the grass roots is the revolt of the young against the old.

Across the border in France we see what happens when changes take place outside the main centre-left party. The results are dramatic, as the Socialist Party now finds itself on the fringes of the political scene.

The problems of the French centre left were evident for some time. The PS had become a bastion of urban, middle-class intellectuals, whose power base was in the public sector unions and the universities. Some of the party’s most prominent figures were quintessential champagne socialists, deeply rooted in the upper echelons of Parisian society. As a consequence, the PS struggled to articulate the wishes and aspirations of France’s working classes, even if it had wished to do so.

When Lionel Jospin, the Socialist candidate in the 2002 presidential election, failed to make it into the second round, defeated by the far-right leader Jean-Marie Le Pen, left-leaning journalists wondered aloud: “Who are these people who vote for the National Front?” Their ignorance was a sign of their disconnection from much of France’s semi-urban and rural voters, many of whom had started voting for the National Front a long time ago.

Middle-class and educated voters have turned towards Jean-Luc Mélenchon, an outspoken and old-school leftist who promises a more authentic embrace of French republicanism and socialism. Until the early 1990s, the French Communist Party (PCF) was a still a commanding presence in French politics. Its then leader, Robert Hue, won more votes in the first round of the presidential elections in 1996 than Benoît Hamon in 2017. The PCF’s cultural shadow – in the form of newspapers and summer festivals – lives on, in part, through Mélenchon’s movement, La France Insoumise. These changes and divisions on the left help explain Emmanuel Macron’s success in shaking up the system. In between the first and the second round of the 2017 presidential election, many leftists argued long into the night about the rights and wrongs of abstention. In the end, roughly a third of Mélenchonistes abstained; the rest voted for Macron in order to defeat the National Front’s Marine Le Pen. In France, renewal takes the form of a choice between the authenticity of the far left and the tempting centrism of Macron’s En Marche! All that is left for the Socialist Party is decline.

****

his swift sweeping away of the centre left is being repeated across Europe. Last year, in the Czech Republic, the ruling Social Democrats were pushed into sixth place in the election, their vote share down from 20.5 per cent in 2013 to just over 7 per cent. The main beneficiary was Andrej Babiš, a billionaire populist, and his “Action of Dissatisfied Citizens” (ANO) movement, which won almost a third of the vote but has since been embroiled in a scandal over its leader’s use of EU subsidies.

The complacency of the old elite on the centre left means that bold changes in leadership can have significant effects. The Labour Party is a case in point. The political fragmentation seen in France and Germany has not escaped British politics. Both the SNP and Ukip absorbed the votes of many who were disenchanted with the status quo. Both parties won significant support from Labour voters.

Unlike Tony Blair – who gleefully enriched himself after leaving office – Jeremy Corbyn, when he emerged from the back benches to contest the leadership after Ed Miliband’s defeat in 2015, was respected, especially by younger voters, for his stolid authenticity. In his first session as leader of the opposition at Prime Minister’s Questions, in September 2015, Corbyn read out letters from the public. His message was clear: I am not here for myself, I am here to represent the people and their concerns. There were many reasons for Labour’s surprisingly strong result in the 2017 general election; the “Corbyn factor” was, of course, one of them.

Elsewhere, on the Iberian Peninsula the left has been renewed by a change in the political class. The ambition of Podemos, a party built out of the “Plaza de Mayo” protests of 2012 in Spain, is to overtake and eventually replace the Spanish Socialist Workers Party (PSOE) as the main party of the left. Podemos has so far fallen short of this ambition and the PSOE has been somewhat revitalised. However, Podemos’s arrival has made the Spanish party system more fragmented and unpredictable. The leaders of Podemos emphasise their distance from the professionalised and often corrupt practices of the traditional elite. University professors dominate the commanding heights of Podemos, but the party’s image is of a down-to-earth citizens’ movement.

Pablo Iglesias, the leader, has a wiry ponytail and an unkempt beard. His politics fuse high-flown ideas with everyday cultural references. He once presented King Felipe of Spain with a box set of Game of Thrones; his message being that the TV series provided the key to understanding the country’s political stalemate.

Yet Podemos has struggled to maintain itself as a “citizens’ movement”. Its base complains that the leaders dominate the party and ordinary supporters have little involvement in formulating policy or in taking key strategic and tactical decisions.

This is unsurprising given how quickly Podemos was propelled into the centre of Spanish political life but it suggests, too, that this new party is not immune from the conflict between office-seeking and vote-seeking that afflicts more established parties. In Britain, it is unclear whether Momentum, the movement behind Corbyn’s rise, is representative of those for whom it claims to speak. Its members are often young, urban and educated.

Do these new leaders or new parties and movements of the radical left represent the birth of a new political project? Or are they themselves symptomatic of the limited reach of contemporary politics, where intense discussions among activists co-exist with deep public disengagement and antipathy towards representative politics?

****

There is another reason, I think, for the puzzling failure of the European social democrats in an age of inequality and it lies with the very project of the left itself.

If we take only the modest vision of social democracy, can this 20th-century attempt to combine economic prosperity with social justice survive in an age of global competition and the integration of markets?

Left-wing politics has at least two core beliefs: in the power of politics over markets; and in the existence of a political agent – the state, parliament, the party – that can exercise that power. If there is no agent, and no domain that we can properly speak of as political, does social transformation enacted by elected representatives make any sense at all?

The view that left-wing policies are powerless against the full force of a globalised economy has been expressed so often that it has become a cliché. But it has never been convincingly rebutted.

And there is no evidence that the European Union can compensate for the weakness of nation states. “Ever closer union” seems to have removed power from national governments but without recasting it at the European level. This leaves any political project relying unequivocally on the power of government to moderate the inequities of capitalism in an uncomfortable no-man’s-land. Stripped of any real political agency capable of enacting its goals, it is stuck in what the German sociologist Claus Offe has called “the European trap”. For Offe, at the EU level the problem is not only of, “What is to be done?”, but also, “Is there anyone to do it?” Far from galvanising political action, recent crises of the EU have had a paralysing effect, especially on the left. Being in a “trap” means being unable to move forwards and unable to move back.

At the heart of the matter is the difficult question of “socialism in one country”. Contemporary parties of the left across Europe simply do not believe that this is possible today, if it ever was. This is partly to do with the legacy of past defeats and partly because of the particular conditions of the global economy. Previous losses loom large in the memory of the left and they inform political action in the present. France is once again decisive, but the trajectory of other left-wing parties, from the Dutch Labour Party to Pasok in Greece, echo the French experience.

François Mitterrand’s election in 1981 came with huge expectations, especially as Thatcher and Reagan were embracing a new era of free markets. His programme of “Keynesianism in one country” promised full employment, reduction of inequalities, nationalisations, the channelling of savings into industrial investment, more support for heavy industry, increased taxes on the highest earners, an expansion of rights within the workplace and a steady reduction in France’s reliance on imported goods. However, on the day he took office, hours before a solemn ceremony in which he placed a rose on the tomb of one of the founders of the socialist movement in France, Jean Jaurès, Mitterrand was being briefed about the perilous state of the country’s foreign currency reserves. He was told that $1.5bn had been withdrawn in a matter of hours and that something had to be done to stop the outflow.

From the very outset of his presidency, the battle raged between political and market imperatives. The key question was whether Mitterrand would defend or devalue the franc. Defending it would require implementing what today we call austerity. Mitterrand doggedly held on until March 1983, when the pressure on the currency became too great. He communicated privately to his prime minister that he wished to take France out of the European Monetary System then, a few days later, changed his mind. After that, as one journalist has put it, the dream gave way quickly to reality.

Policy defeats such as this define the history of the European left since the 1980s. Most recently, Greece’s ruling Syriza experienced the same asphyxiation of a left-wing project but in an accelerated fashion. In January 2015, jubilation and hope at the arrival of a far-left government gave way, six months later, to despair as the same government signed a new bail-out agreement with the EU, which was even tougher than what Greeks had rejected in a referendum.

However, the shadow of past defeats for the European left are not responsible on their own for the belief that socialism in one country is impossible. The actual external constraints in place today are more powerful than ever before. These constraints were originally the consequences of the early policy defeats. Today, they have become one of the causes of the difficulties in pursuing firmly left-wing programmes.

After his turn towards austerity, Mitterrand covered his tracks by relaunching European integration. The Single European Act of 1986 promised that national growth would come from increases in cross-border trade rather than expansionist Keynesian policies. Aware that short-term increases in public spending were always tempting, especially for parties of the left, governments opted to strengthen external constraints.

The 1990s saw a proliferation of fiscal and monetary rules. Embraced by many governments of the left, these aimed at introducing rigour and discipline into national macroeconomic policies. Not all such constraints were at the European level: the incoming Blair government of 1997 made its commitment to fiscal prudence a central part of its campaign manifesto.

The high point of this period was the creation of the euro, the disciplining instrument par excellence that took away the ability of governments to set their own interest rates. “Ever closer union” came to mean ever deeper forms of regulatory harmonisation, across a wide variety of sectors, limiting the ability to pursue a path different from one’s neighbours. Meaningful policy divergence on the core question of income redistribution is more difficult – practically and constitutionally – than in the past, particularly as long as one remains a member of the single market and the eurozone.

For the European left, this presents a new dilemma: it must decide at what level to operate, the national or the pan-European. If national possibilities have been snuffed out by the proliferation of pan-European fiscal rules and monetary constraints, then perhaps the right terrain to act upon is Europe at large. This is the view of the SPD leadership in Germany and it is a common belief across the European left. Advocates of it point to the EU’s growing appetite for progressive causes: cracking down on tax avoidance by tech giants such as Google, Amazon and Apple, moves to introduce a financial transactions tax and so on.

However, the opposite view is also gaining traction. There is burgeoning interest among economists in the mechanics of parallel currencies and how the use of tax bonds may help countries leave the eurozone. The hope is that eurozone exit would create more space for macroeconomic policy divergence. In many senior policy circles in Europe, it is simply no longer a radical position to argue for the dismantling of the eurozone or for the exit of some of its members.

or the left, these discussions form the basis of very different and rival projects. Political energies can either be channelled at the national or the European level, but not both, as one will necessarily undermine the other. Opting for the EU level poses a number of problems. Policy co-ordination – difficult enough at the national level, where central banks, finance ministries and powerful interest groups find ways to water down policies – is infinitely more difficult. In a union of 27 member states, governments are, at any one time, of different political outlooks.

Also, if national political parties on the left struggle to maintain close ties to their own voters, at the European level the sense of disconnection and isolation is even greater. There is a pan-European political life but it is thin and patchy. Recent attempts to create transnational lists for next year’s European parliamentary elections failed miserably: the European Parliament voted against the proposals and a petition in support of it collected around 35,000 votes, a tiny fraction of the total EU population of just over 500 million.

Historically, the left as a political force emerged from the confrontation between the wishes and desires of European workers and the inability of 19th-century laissez-faire liberalism to satisfy them. The incubator for left-wing ideas has long been the gap between what people want and what the social structure of capitalism can deliver.

Today’s parties of the left in Europe tend to be too socially deracinated and isolated to know what it is that people want. And given the complacency of many of the leading figures, it is not clear they are interested in finding out. The degree to which new movements are representative of their societies is crucial if they are to be a path for the renewal of the left. And so, the question for the political parties of the left is how they can reconnect with societies marked by intensifying individualism.

The left in Europe should not look back nostalgically at the “golden age” of social democracy and ask how that can be recreated. The world of the 1940s or 1950s was very different, as were the early years of the 20th century, when the emergence of mass politics was transforming political systems and giving rise to new social movements. Nor should the left hope to recreate this golden age at the European level, a wish that is more utopian than ever today.

The challenge of the present is to combine a renewal of the theoretical tools of left-wing social and political thought with the discovery of what people are thinking about and what it is that they want. And then to build a new political project at the national level, based on what changes to society would be required in order for those needs and desires to be met.

Chris Bickerton is reader in modern European politics at the University of Cambridge and author of “The European Union: A Citizen’s Guide”(Penguin)