Among the old stalwarts of the centre-left, there is a simple explanation for the decline of the parties they used to lead: immigration. In recent interviews with the Guardian, Hillary Clinton, Tony Blair and the former Italian prime minister Matteo Renzi all sounded the same note, declaring that Europe must “get a handle on migration” to stop rightwing populism. Hardly a week passes without some candidate or columnist declaring that liberals will only regain power when they lock down the borders.

The obsession with immigration is not an accident. It reflects a widely held belief that the decline of the grand parties of the centre-left across Europe – the Socialist party (PS) in France, the Democratic party (PD) in Italy, the Social Democratic party (SPD) in Germany – has been caused by the rise of the new parties of the populist radical right, who have “stolen” the old working-class vote with a nativist, even authoritarian, message. Consequently, centre-left politicians have been scrambling to come up with policies to “win back” the working class. (Blair was already worrying about this in 2001, according to his former EU adviser Stephen Wall, who recalled the prime minister saying: “The one thing that could lose me the next election is immigration.”)

This view is hardly confined to Clinton and Blair. In the wake of Donald Trump’s unexpected victory, many American liberals insisted that “winning back the rust belt” could only happen if the Democrats embraced white fears about immigration. Indeed, Clinton herself had a long track record of statements and Senate votes against “illegal” immigration and for “border security” (including a “fence”), while prominent European social democrats have been calling for “immigration realism” since the late 2000s.

But since the so-called refugee crisis of 2015, these worries have escalated into a panic, as the leaders of Europe’s social democratic parties scramble to show their concerns over immigration. In the last year, the leader of Belgium’s Socialist party has issued the blanket declaration that “migration to Europe must decrease”, while the new leader of Germany’s SPD has said that her country “cannot accept all” asylum seekers that arrive at its borders. The Danish Social Democrats have gone so far that their statements on immigration are hard to distinguish from the far-right Danish People’s Party – accusing Muslims in Denmark of living in “parallel societies”, and arguing that immigration “undermines” the Danish welfare state.

This dramatic shift in the rhetoric of ostensibly centre-left parties is part of a larger panic over how to halt the spread of rightwing populism across the west in recent years. The conventional wisdom has been largely steered by a growing group of academics and pundits, often of the right or centre, who offer the same advice: social democratic parties will perish unless they take care of the “left behind” voters by limiting immigration. Some academics now even go so far as to openly defend white identity politics.

The argument that a tougher stand on immigration will revive the social democratic parties – and arrest the rise of the radical right – is based on two basic errors, which together reflect a larger misunderstanding about the historic role of centre-left parties.

The first mistake is the widespread assumption that the rise of rightwing populism and the decline of traditional centre-left parties are two sides of the same coin – both caused by working-class voters abandoning the old social democrats for the nativist message of the new populist radical right. The second misperception, closely related to the first, is that the voters who now support the populist radical right are largely the white working class that used to vote reliably for social democratic parties.

As the data shows, both of these widely repeated assumptions stand on loose empirical footing. In fact, most populist radical-right voters are not working class, and the majority of the working class does not support the populist radical right.

These errors are based on a larger misunderstanding about the history of social democratic parties. Social democracy is an ideology that supports egalitarianism and social justice through the framework of liberal democracy and a mixed economy. Inspired by the Marxist concept of class struggle, social democracy aims to uplift all marginalised groups. But those who argue that centre-left parties need to pander to white anxiety about immigration are essentially saying that social democratic parties are first and foremost an interest group for “the working class” – which is always, in these accounts, assumed to be white.

This misdiagnosis of the decline of the centre-left – and the rise of the populist right – leads to the wrong prescription for reviving social democracy. In fact, centre-left parties have been trying to “act tough” on immigration for decades, and have often supported policies to limit immigration, but it has not prevented their decline.

At least since the beginning of the new century, a debate has been raging about how to respond to right-wing populism – largely between those who regard it as the byproduct of “economic anxiety”, and those who see it as a form of cultural backlash. But both sides have the prescription wrong: if social democracy is to survive, its politicians need to return to their core values – rather than chasing a mirage that looks like their former core voters.

The key to reviving the fortunes of social democracy is not to pander to the nativism of part of the white working class, but to embrace the ideas and policies that are fundamental to social democracy – egalitarianism, social justice, solidarity, the right to social protection and a comprehensive welfare state. These values represented a widely shared common sense for the vast majority of Europeans in the second half of the 20th century – before their hegemony was eroded by three decades of neoliberal ideas and policies. The only way back for social democracy is to fight to make these values dominant once again.

At first sight, it makes sense to link the decline of social democratic parties to the rise of populist radical-right parties. But correlation does not always equal causation. First of all, they did not happen at the same time. More importantly, they have separate causes.

If we look at the average vote share for social democratic parties in western Europe, we can see that it rose to more than 30% in the 1950s and remained stable until the late 80s. In the late 90s, the average vote share fell back to just under 30%, only to drop off sharply in the 2000s. Today, it is slightly above 20%.

Until the beginning of the 1980s, populist radical right parties were largely irrelevant in western Europe, polling at about 1%. By the 90s, this had increased slightly to about 5%. But, as the vote share for social democratic parties fell in this decade, populist parties did not grow. In the 2000s, they started to increase again, albeit modestly, averaging roughly 10% today.

These figures represent the average of votes across western Europe, but looking at specific countries makes it even clearer that the rise of the populist radical right has not caused the decline of social democratic parties. In many cases, the populist radical right has risen without any comparable decline for the centre-left; in other cases, the centre-left parties began their sharp decline long before there was any major populist radical right party.

In Switzerland, western Europe’s most successful populist radical-right party, the Swiss People’s party, almost doubled its support between 1995 and 2015, from 14.9% to 29.4%, while the Social Democratic party of Switzerland lost a mere 3% in that same period. The German SPD started its decline after 1998, but took its biggest hit in 2009, four years before the populist radical-right Alternative für Deutschland (AfD) was even founded.

The Dutch Labour party (Partij van de Arbeid, or PvdA) took a hit from the rise of the rightwing populist Pim Fortuyn List in 2002, but recovered well the next year. Its full implosion in 2017, losing 19.1% (more than three-quarters of its 2002 total), was several years after the highpoint of Geert Wilders’ Party for Freedom (PVV), in 2010. Moreover, the combined far right, ie the PVV and the new Forum for Democracy (FvD), made only relatively moderate gains (+4.9%) in 2017. Not to mention Spain’s Social Democratic party, who were almost halved before the recent rise of the populist radical right Vox party, and actually achieved a significant victory, going from 22.6% to 28.7%, in the same election that Vox entered parliament with 10.3% of the vote.

There is a simple explanation for this. The decline of social democratic parties has largely different causes from the rise of populist radical-right parties. First and foremost, the decline of social democratic parties has mainly been caused by the transformation of an industry-based economy into a service-based economy. This has led, among other things, to a sharp decline in traditional working-class jobs and a relative decline of all working-class people within the broader population.

Confronted with a declining working class and a growing middle class, social democratic parties started to target the latter at the expense of the former. Inspired by Bill Clinton’s successful move to the centre in the 1992 US presidential elections, Tony Blair rebranded the Labour party as New Labour in 1994, and embraced a new “integration consensus” centred around cultural (multiculturalism), economic (neoliberal globalisation) and national (EU) integration.

Moreover, he tried to “depoliticise” politics – claiming to propagate a new “pragmatic” approach and “common sense” solutions, in which everyone was claimed to be a winner. Soon other western European social democratic parties would follow, such as the Dutch PvdA and the German SPD, whose leader Gerhard Schröder claimed to represent “the new centre” (“die neue Mitte”). Slowly but steadily, even the the French socialists and the various southern European parties followed suit.

Facebook Twitter Pinterest European far-right leaders in Koblenz, Germany, from left: Matteo Salvini (Lega), Frauke Petry (AfD), Geert Wilders (PVV) and Marine Le Pen (National Rally). Photograph: Wolfgang Rattay/Reuters

During the 1990s, the average vote share for social democratic parties only fell back a small amount – but the populist radical right parties did not rise further at all. The big transformation came only in the 2000s, most notably in the wake of the terrorist attacks of 9/11. The political framing of the “war on terror” helped to make cultural flashpoints like immigration, Islam and security the predominant issues of the new century. In many western European countries, this new political alignment – defined in cultural rather than economic terms – resulted in the Greens becoming the main leftwing party, and the populist radical right becoming the main rightwing party.

Leaders of the populist radical right like to be seen as the successors of the old social-democratic “workers’ parties” – as the authentic representatives of “real working people”. For instance, the former party leader of the AfD, Frauke Petry, openly declared that “the AfD wants to become the new SPD”. And the self-appointed far-right ambassador-at-large, the former Trump adviser Steve Bannon, claims that “there’s an awful lot of guys on the [Bernie] Sanders side who are going to come in and vote for us in 2020”.

But even this ambition involves a misleading assumption about the history of the working-class vote. It is true that, since the second world war, the working class in western Europe has voted disproportionately for social democratic parties, and communist parties where relevant – but a significant part of the working class has always voted for rightwing parties.

Since the emergence of the populist radical right in the early 1980s, its voter base has continually shifted and evolved – with white workers as an important but far from dominant component. The few populist radical right parties of the 80s had small electorates, polling under 5%, and their voters had a variety of class backgrounds and past voting behaviours. Parties such as the Belgian Vlaams Bloc and the French Front National (FN) were particularly strong among the self-employed, but also attracted both blue-collar and white-collar workers. Unsurprisingly, given that populist radical-right parties declared themselves an alternative to both left and right, they drew voters from both sides as well as picking up some first-time and non-voters.

When some populist radical-right parties began to post better results in the 1990s – between 5% and 15% of the vote – there was also a visible shift in the composition of their supporters. Those parties held on to their predominantly middle-class supporters, but began to add significant votes from among the working class. Hans-Georg Betz, one of the pioneers of the academic study of populist radical-right parties, was the first person to note what he called the “proletarisation” of the populist radical right.

The perception that the working class was shifting its allegiance to the populist radical right in the late 1990s was further strengthened by a shift in their propaganda. Previously, many of these parties had backed typically neoliberal policies such as lower taxes and privatisation, but now they began to strongly support a chauvinist welfare state – one whose benefits would be robust, but only for “our own people”.

As a consequence, the electorates of the more successful populist radical-right parties, like the French FN and Austrian Freedom party, started to more closely resemble, in terms of class structure, those of traditional socialist and social democratic parties in the 1990s. In both popular and academic writings of the turn of the century, these successful populist radical-right parties began to be called “workers’ parties” – because they were among the more popular parties with the (white) working class. The stereotypical populist radical-right voter in western Europe was now depicted as a young(ish), lower-educated, working class male.

But this was a misleading picture: young and working-class voters were indeed overrepresented within the electorates of successful populist radical-right parties, compared with their proportions in society as a whole – but they still constituted at best a plurality, not a majority, among those parties’ electorates.

Facebook Twitter Pinterest The chair of Germany’s SPD, Andrea Nahles, this month. Photograph: Ronald Wittek/EPA

In fact, most voters for populist radical-right parties were not working-class – and most working-class voters did not vote for the populist radical right. A recent study found that “only” 31% of “production workers” and 23% of “service workers” voted for west European populist radical right parties between 2000 and 2015. And while the FN and Austria’s Freedom party are exceptions – with workers constituting 45% and 48% of their electorates, respectively – the figures are much lower for other such parties, with Italy’s Lega Nord at only 17%, for example.

Furthermore, surveys show that populist radical right parties do not primarily take voters from social democratic parties. And voters who abandon social democratic parties do not primarily move to populist radical-right parties.

In the 2017 German parliamentary elections, the SPD, which slipped from 25.7% to 20.5%, lost more voters to each of the other mainstream parties, and to non-voting, than to the AfD. And the AfD, which entered the Bundestag for the first time with 94 seats (12.6% of the vote), won more votes from Angela Merkel’s centre-right Christian Democratic Union, as well as from non-voters and “other parties” (notably the extreme right National Democratic party), than from the SPD.

A similar pattern could be seen in the Netherlands and Italy. The Dutch PvdA was wiped out in 2017, falling from 25% to 6% – but its former voters largely shifted to the green, radical left, and social-liberal parties, rather than the populist radical right FvD and PVV. In Italy the following year, Matteo Renzi’s Democratic party lost power – but almost none of its voters shifted to Matteo Salvini’s Lega Nord, which mainly took votes from other rightwing parties and first-time voters.

What these examples demonstrate is that the recent growth of populist radical-right parties – unlike their initial expansion in the 1990s – has not been driven by winning over more of the working class. The real story is that responses to events such as 9/11 and the “refugee crisis” by mainstream commentators and politicians brought the arguments of the populist radical right more into the mainstream discussion – and their “solutions” consequently became acceptable to broader sections of the public. As a result, the most successful populist radical-right parties now are Volksparteien – “people’s parties”, rather than “workers’ parties” – and do not represent just the working class.

This is not simply an academic debate. These misconceptions about populist radical-right voters have had serious consequences for centre-left politics, because they have led many social democratic parties to pursue failed strategies against the populist radical right.

As it became clear that the “third way” had merely postponed the electoral decline of social democratic parties, a search for alternatives began. But the overwhelming strategic focus was on how to “win back” the “abandoned” (white) working-class vote.

This argument has been dominated by two rival camps – crudely, liberals against socialists. Each camp has a different view of the main cause of the decline of social democratic parties and the rise of the populist radical right, but they share the notion that “winning back the working class” is the most important task.

One version of this debate took place in the US after Trump’s victory: did the (white) voters of Michigan and Ohio defect to Trump because they were suffering financially (“economic anxiety”), or was it for reasons of racial and ethnic supremacy (“cultural backlash”)?

Liberals tend to regard (white) working-class support for the populist radical right primarily in terms of cultural backlash, although they do not deny that economics has played a role. This is correct – as we shall soon see – but their responses have been ineffectual, often taking the form of vague calls to “reclaim nationalism”, which almost always come down to tightening immigration.

While many liberals have been at the forefront of attacks on “populism” as today’s biggest threat to democracy, many prominent liberals have implicitly accepted the framing of the populist radical right. Sigmar Gabriel, who led Germany’s SPD for nearly a decade, declared in 2016 that the lesson of Clinton’s loss to Trump was that “those who lose the support of workers in the rust belt cannot be rescued by the hipsters of California”. This isn’t too far removed from the simplistic populist dichotomy between “somewheres” and “anywheres” popularised by the writer David Goodhart.

For liberals such as Tony Blair, whose eponymous Institute for Global Change has been at the forefront of pushing a “progressive approach” to immigration, the populist backlash is not because of neoliberal globalisation, and the massive economic inequality it created within previously fairly equal societies in western Europe, but about “the things that people feel are damaging about European immigration”.

It is a familiar story, shared by both social democrats and the populist radical right: social democratic parties supported open borders against the wishes of “the people”, which usually means the (implicitly white and nativist) working class. For decades populist radical right politicians have claimed that social democratic parties have “betrayed” the white working class in favour of immigrants and Muslims, their alleged new electorate. In reality, social democratic parties have a much more complex relationship with immigration and multiculturalism.

Most social democratic parties were indeed vocal supporters of immigration and multiculturalism, particularly during the 1980s and 90s. In many western European countries the links between the parties and the anti-racist movement were tight – such as in France, where the first president of SOS Racisme later became first secretary of the PS – and social democrats were at the forefront of the political struggle against the far right.

But social democratic parties across western Europe were often less supportive in their actual policies. They did not accept or acknowledge that their states had become “immigration countries” like the US or Canada – and they implicitly or explicitly supported restrictions of both economic immigration (for example, in the wake of the oil crisis of the early 1970s) and political asylum (for instance in the wake of the Yugoslav civil war in the early 90s).

Similarly, their support for “multiculturalism” was mainly symbolic – as no party developed a coherent policy, let alone a social democratic vision on multiculturalism. Blair’s own legacy is a good case in point. As Will Sommerville of the Migration Policy Institute has argued, Blair and New Labour expanded economic immigration but publicly championed a “stronger, more restrictive approach to asylum and security”.

In many ways, Blair’s approach is illustrative of the confused way social democratic parties responded to the increasing politicisation of immigration during the 1990s. With the third way no longer arresting their electoral decline, centre-left parties joined the centre-right in a strategy of co-optation – in other words, embracing the issues of the populist radical right in an attempt to marginalise the parties espousing them. The call for “more realistic” immigration policies increased in the wake of the terrorist attacks of 9/11, as did calls for “tougher” integration policies for (mostly) Muslim “immigrants. Just remember Gordon Brown’s “British jobs for British workers” speech in 2007, a clear dog-whistle to the populist radical right – the National Front had campaigned with a similar slogan in the 1970s.

The left alternative to this faltering liberal approach has come from those who commonly identify as “democratic socialists”. This camp has openly embraced the idea of “left populism”, heavily influenced by Chantal Mouffe, a Belgian political theorist based in London, and her late husband, the Argentine Ernesto Laclau. They are most famous for their critique of the third way aspirations to eliminate conflict and partisanship from politics – and their call for a post-Marxist “radical democracy” has influenced a whole generation of leftist academics and politicians, particularly in Latin America and southern Europe.

Since Laclau’s death in 2014, Mouffe has become the most visible and vocal torchbearer of a new left populism, most recently in her aptly titled book For a Left Populism. She uses the terms “left” and “populism” very broadly, however, defining neither very clearly. Her key argument is that the left must disrupt the neoliberal consensus by “repoliticising” politics – accepting that strife and conflict between opposing groups are fundamental to political life. Unlike orthodox Marxists, Mouffe argues this conflict should not unfold along traditional class fault lines – or any specific fault lines at all. Instead, she argues that the left must unite around a deliberately vague anti-establishment programme, designed to accommodate the broadest possible coalition of progressive causes and grievances.

Mouffe and many other supporters of this left populist strategy saw the rise of a new progressive dawn with the electoral success Podemos in Spain and Syriza in Greece a few years after the economic crisis – she even published a book with one of the leaders of Podemos, Íñigo Errejón. But since then, Syriza has broken the hearts of left populists across Europe, succumbing to EU pressure, while Podemos has lost momentum as a consequence of personal and political conflicts. Mouffe has now put her hope in Jean-Luc Mélenchon, the French social democratic insider who reinvented himself as a left populist outsider.

Facebook Twitter Pinterest Íñigo Errejón, one of the leaders of Podemos, campaigning during regional and European elections in Spain. Photograph: Victor Lerena/EPA

Although Mouffe stays away from the nativism lite of some other left populists – most notably Sahra Wagenknecht and her new movement Aufstehen (Stand Up) in Germany – she also clearly targets the white working-class voters, particularly the ones the third way lost to the populist radical right. In several interviews Mouffe has said: “When citizens go to vote they see no difference between the choices facing them. That has allowed the development of right-populism. Marine Le Pen speaks to the pain of the popular classes, telling them that foreigners are the cause of their problems. We need another, opposed discourse built on the basis of equality.”

The left populists share the assumption that the (white) working class votes for the populist radical right out of economic anxiety rather than cultural backlash. Hence, once the left provides them with a better socio-economic alternative, they will no longer care about Islam and Muslims.

However, as the American political sociologist Seymour Martin Lipset demonstrated more than 60 years ago, some part of the working class is authoritarian and nativist. Decades of research have since confirmed that these nativist working-class voters – like the other voters who support the populist radical right – are first and foremost motivated by opposition to immigration (and Islam). In other words, by cultural backlash rather than economic anxiety.

But while “liberal populists” like Blair or Clinton may have a better understanding of what drives the voters of the populist radical right than the left populists, their proposed solution to the problem has only helped make it worse. Academic research consistently shows that when mainstream parties move to the right in an attempt to co-opt the issues of the radical right, it does not hurt populist right parties – in fact, it often helps them. Moreover, other research shows that it does not stop the electoral bleeding of social democratic parties either.

This makes perfect sense. By prioritising immigration as an issue – and reinforcing the negative depiction of migrants and migration – mainstream parties only help to boost the main issue and frame of the populist radical right. Moreover, populist radical right voters are not only nativist, they are also populist, which explains why the “immigration realism” of social democratic parties is ultimately not effective. Even when they go further than mere rhetorical support for tougher immigration and integration policies, most (working-class) voters who oppose immigration and believe it to be a salient topic also believe that mainstream parties in general, and the social democratic party in particular, cannot be trusted. In fact, their rightward turn is taken as confirmation that a vote for the populist radical right is the most effective choice, as it ensures that other parties implement their policies.

To regain relevance in the 21st century, social democracy has to go beyond appeals to populism – in both its liberal and left forms. The surprising successes of Jeremy Corbyn in the UK and Bernie Sanders in the US have already shown that there is an appetite for a revival of social democracy.

But before social democracy can be rejuvenated and made ready for the 21st century, it has to come to terms with its own role within the political model the populist radical right now so openly defends. While social democracy never openly defended white supremacy, its political system, and welfare state, was built upon a social system that preferred white native workers over non-white “guest workers” – and embraced the patriarchal dictates of single-income male-dominated family life. While social democratic parties, and trade unions, were quick to appeal to non-white workers, they did not fundamentally change their basic models.

Social democracy needs to reassert its ideals in a way that is inclusive of all workers. It should return to the theory rather than the practice of European social democracy – an egalitarian ideology based on solidarity with all socially weaker groups and individuals, irrespective of class, race, or sexuality. In the early 21st century, throughout western Europe, a growing percentage of the shrinking working class will be female and non-white (or of immigrant descent).

Moreover, by today’s standards, the traditional factory worker has a relatively privileged position, a well-protected and well-rewarded permanent job. In a world of neoliberal globalisation, not only are many of these classic working class positions outsourced to poorer countries, they are replaced by precarious positions that lack protection and security. It is the great challenge of social democracy in the 21st century to find a way to integrate the so-called “precariat” into a broader movement for economic and social justice. Whether this can still be done under the traditional header of “the working class” is of secondary concern. What is more important is that the identity is built on socioeconomic interest rather than ethno-national identity.

It is important to stress that what we are facing at the moment is a crisis of social democratic parties – not a crisis of social democratic ideals. It might seem delusional to make this claim at a time of seemingly never-ending electoral defeats of social democratic parties, but social democracy is alive and well. In fact, I cannot remember another period in my adult life where there is such a vibrant debate on the future of social democracy. To be fair, this debate is, ironically, almost exclusively in the US, and under the mostly erroneous term “democratic socialism”, but it is raging, and not just in the traditional progressive “salons”.

Many traditional social democratic policies are supported by the majority of the populations in Europe and North America. A recent OECD study, for example, found that a majority across all 21 member states support higher taxes for the rich to support the poor. More than half of the people want their government to ensure better pensions (54%), while almost half feel the same with regard to better healthcare (48%). More than one-third (37%) even support a guaranteed basic income benefit.

And yet social democratic parties continue to lose electoral support, and the welfare state – an institutional expression of many of these policies – is consistently dismantled. This is because policies such as higher taxes for the rich and a strong welfare state are grounded in a social democratic ideology, which has lost its ideological hegemony to neoliberalism since the 1980s.

Neoliberalism is not just an economic system, but also an ideology, which has fundamentally transformed the way people see politics. In fact, one could argue that neoliberalism has been more successful as an ideological project than an economic programme. Active citizens have become passive customers, who consider the public sector an inefficient alternative to the private sector – and regard competition, between firms or individuals, as the best model for an ideal society.

Consequently, before there can be any electoral revival of social democratic parties, social democrats need to challenge the assumptions of the neoliberal society, and re-establish their own ideas of egalitarianism and solidarity as the new common sense. As the Italian Marxist Antonio Gramsci explained almost a century ago, political success can only come after cultural hegemony is established. Only when people support the underlying values of social democracy can parties successfully campaign on social democratic policies. Moreover, without broad support for key social democratic values, even popular policies can easily be defeated.

‘We the people’: the battle to define populism Read more

The project of re-establishing the cultural hegemony of social democratic ideas is going to require the mobilisation of people outside of the existing political parties, including the social democratic ones. In most countries, these parties are anyway run by people who joined during the height of the third way, which they erroneously consider to be the height of “real” social democracy. Similarly, ideological rejuvenation should happen in collaboration with, but independent from, Green and “radical left” parties, which have overlapping but fundamentally different ideological projects.

The revival of social democracy will require a new cultural and political infrastructure, centred, at first, outside of electoral politics. It should include the trade unions, which, despite weakened membership and power, still have better connections to working people. It should include progressive minority organisations, particularly those focused on socioeconomic concerns, and new grassroots organisations, rooted in local communities.

In short, reviving social democracy will require a new social democratic movement – one that is bigger, bolder and more energetic than the existing parties.

• Follow the Long Read on Twitter at @gdnlongread, and sign up to the long read weekly email here.