For a number of years Europe has been in the midst of a significant challenge from national populism, as a succession of recent elections have shown in Italy, Austria, Hungary and Sweden. Yet this is a movement that remains poorly understood. Parties on the radical left and Greens are also making gains in some countries, but they are having nothing like the electoral or policy impact of the far right. It has emerged in democracies that were always thought to be immune to this political force. When I first started working on the subject in the late 1990s, an unwritten law of sorts was that there were four democracies that would never succumb. They were Sweden and the Netherlands, because they were historically liberal, the UK because of its strong political institutions and civic culture, and Germany, because of the stigma left by the events of the second world war.

Today’s thinkers, writers and groups on the left have subscribed to a number of theories, all of which are incorrect

But fast forward only 20 years, and each of those countries has now experienced a major populist rebellion. Pim Fortuyn and then Geert Wilders in the Netherlands. The Sweden Democrats, who recently reached a new record share of the vote. Alternative for Germany, which has more than 90 seats in the Bundestag and seats in 15 of Germany’s 16 state parliaments. And in the UK, Nigel Farage and the UK Independence party forced a referendum on Britain’s EU membership which voted for Brexit. Sometimes we forget how quickly radical change in politics can occur.

The left has always struggled to make sense of national populism which seeks to prioritise the culture and interests of the nation, and promises to give voice to a people who feel that they have been neglected, even held in contempt, by distant and sometimes corrupt or self-serving elites. And today’s thinkers, writers and groups on the left have subscribed to a number of theories, all of which are incorrect. They claim this volatility is simply a shortlived backlash against something – whether immigrants or “the system” – rather than a positive vote for what national populists are offering, not only more restrictive immigration policies but also a more responsive political system and more equal economic settlement.

Another misconception, building on Marx, is that the likes of Donald Trump, Marine Le Pen or Matteo Salvini are driven by people’s concerns about economic scarcity, competition over wages or jobs, and, particularly today, by the effects of the post-2008 financial crisis and austerity. A third is the mistaken belief that all these awkward and troubling movements are essentially a reflection of lingering racism in society, and perhaps even latent public support for fascism. Others argue, again wrongly, that voters are being ruthlessly manipulated into voting for the populists by dark and shadowy right-wingers who control the media or big tech.

These ideas are not mutually exclusive, but they have dominated much of the left’s thinking about populism. Yet there isn’t much evidence to support any of them. Clearly, only a fool would argue that things like the financial crisis, social media and racism are not important. But they have been given a level of influence in the debate that is wholly disproportionate to their significance, and they distract from dealing with the actual grievances that are fuelling the rise in populism.

The idea that Brexit can be explained away via references to big tech firms, that Trump is merely a byproduct of racism, or that the dramatic political shifts in Europe can be resolved by redistribution and tackling inequality, are mere comfort blankets. What we need to focus on instead is how in most western democracies the rise of national populism has coincided with the fall of social democracy. National populism has recognised how the foundations of politics are moving while the left, for the most part, has clung to outdated theories.

Play Video 1:29 Jair Bolsonaro's provocative views in six clips – video

The current wave of national populism actually began decades ago, in the late 1970s and 80s, a “backlash” to the 60s liberal revolution that never truly went away. Since then it has been most successful in some of the most prosperous and stable economies, including those with strong rates of growth and low unemployment.

Even in Britain we conveniently ignore the fact that Farage and his self-anointed People’s Army first enjoyed major success at the 2004 European parliament elections, after 48 consecutive periods of economic growth, and drew much of their early support from affluent conservatives (it was only later when Ukip became more successful among blue-collar workers). The tendency to dismiss these movements as a political home for old, white racist men ignores the fact that Le Pen picked up much of her support not only from young men but young women in France, while in Austria, Germany, Italy and Sweden, national populists are strongest among the under-40s or draw their support fairly evenly from across age groups. And, when it comes to racism, studies have shown that this is falling, not rising.

So what is really going on? National populism is revolving around four deep-rooted societal shifts: the “four Ds”. First, there are high levels of political distrust, which are being exacerbated by populist leaders who paint themselves and their followers as victims of a political system that has become less representative of key groups. Second, many people have strong and entrenched fears about the perceived destruction of national cultures, ways of life and values, amid unprecedented and rapid rates of immigration and ethnic change. Accompanying this distrust and fear are anxieties related to deprivation and the loss of jobs and income, along with a strong sense that they and their ethnic and social group are being left behind relative to others in society.

Finally, many political systems in the west are having to grapple with a new era of dealignment, in which bonds between voters and traditional parties are breaking down, and hence the path for new political challengers is much more open.

When you take a closer look at these four currents it becomes abundantly clear that there is nothing ephemeral about national populism, and we will be living in an era of heightened volatility for many years to come.

• Matthew Goodwin is co-author of National Populism: The Revolt Against Liberal Democracy