On Friday 21 November, the Conservatives and Labour woke up with quite a hangover as the UK Independence Party (Ukip) toasted another by-election victory, this time in Rochester and Strood. Yet the problems faced by David Cameron and Ed Miliband go far deeper than the present failings and struggles of their two parties.

It cannot be that all mainstream parties in western Europe are led by incompetent politicians. However, all the countries – even Angela Merkel’s Germany, where the anti-euro party the Alternative for Germany is rising – face similar problems.

The characters change but, essentially, the plot remains the same. The old order is being thrown out. Populists of a leftist, rightist and nationalist bent are thriving. It is a process that goes far beyond a few personalities. It is about the rejection, by swaths of the electorate, of a way of ­doing politics. More and more people feel ignored and increasingly powerless – and contemptuous of elites.

In his seminal book Ruling the Void, published posthumously in 2013, the political scientist Peter Mair proclaimed: “The age of party democracy has passed . . .” Before his death in 2011, he foresaw “an emptying of the space in which citizens and their representatives interact”. Mair observed politicians and the electorate becoming increasingly indifferent to one another, with citizens “withdrawing and disengaging from the arena of conventional politics”. Since then, the old political order has floundered not only in the European elections but in national ones, too. Mair’s analysis seems ever more compelling.

The rise of insurgent parties and groups is the result of something far more profound than the eurozone crisis, austerity and the tendency to scapegoat “outsiders” during economic turbulence. As Mair recognised, discontent with mainstream parties has been brewing for decades. The trends that have led us to despair of the health of Westminster politics – the fall in electoral turnout and party membership and the rise in electoral volatility – are detectable in all of Europe’s leading democracies. Nearly 80 per cent of the elections with the lowest turnouts in western Europe between 1950 and 2009 occurred after 1990. Sixty per cent of the most volatile elections – those in which the most votes changed from one election to the next – were also in this period.

Politics has become more unpredictable. The idea of loyally supporting a political party, as one might a football club, is becoming archaic. Tribal and class loyalties no longer determine voting decisions. Trade union membership and church attendance have hugely declined and, a generation after the end of the cold war, few regard their choice of political party as an important part of their individual and collective identities.

Across Europe, mainstream parties are discovering that their core support is weaker than they imagined. In Germany, the two main parties received 87 per cent of the vote in 1983. By last year, that had fallen to 67 per cent, despite an upswing for the Christian Democratic Union under Merkel’s leadership. In Ireland, Fine Gael and Fianna Fáil received just 56 per cent of the vote in the last national election, in 2011, compared to 85 per cent in 1982. And, in Austria, the two leading parties got 93 per cent in 1979, but only 51 per cent last year.

Though much is made of the collapse in Conservative and Labour Party membership – the Tories had more than 2.5 million and Labour a million in the 1950s; both have under 200,000 today – this is merely an extreme example of what is happening throughout Europe. As Mair observed, between 1980 and 2009 party membership fell by at least 27 per cent in 13 western European democracies, and by 66 per cent in the UK. Today, the average party membership rate in Europe is 4.7 per cent, compared with almost 15 per cent in the 20 years after the Second World War.

As the membership of established parties has declined, a lack of activists on the ground has restricted their ability to campaign effectively, creating an environment that is more hospitable to new parties. It has also had a deleterious effect on the character of political parties. The combination of fewer card-carrying members and more career opportunities in politics means that “the proportion of believers is likely to shrink while the proportion of careerists is likely to grow”, as the political scientist Ingrid van Biezen has written.

After the financial crash of 2008, many believed, including Ed Miliband, that an economic crisis caused by the collapse of the banking system would reinvigorate the left. This judgement informed Miliband’s decision to run for the Labour leadership and his approach in the role. But while anger at in­equality is a potent force, it has not translated into gains for the mainstream left at a time of austerity. The crisis of the European left can be traced not just to austerity, but also to the lack of options it is seen as offering. The mainstream left has embraced open borders and mass immigration as well as the increased power of the European Union, in the face of opposition from the public.

Mainstream parties have accepted that their power is limited by larger forces in the modern world. When Tony Blair said that politics was no longer about using the “directive hand” of government, he was accepting limits on what he could achieve. As the parties of centre left and centre right have accepted these constraints, so the difference between them has come to seem insignificant: a study found that the ideological difference between Tory and Labour election manifestos since 1997 has been only a third as large as between 1974 and 1992.

Feelings of alienation have fuelled discontent with politics. The political scientist Michael Laver has argued that “vote-­seeking parties may make voters miserable”, by making them feel that the leading parties do not represent their views on issues such as immigration and the economy. This process is exacerbated by the “professionalisation” of the political class. Candidates’ backgrounds have become increasingly similar. In the case of the UK, parties have imported techniques from the United States to target swing voters rather than their core support.

In the absence of significant ideological differences between the leading mainstream parties, elections have become more about the personalities of individual leaders. This has been a boon to challenger parties throughout Europe. And it has reduced the barriers to entry for any new party.

In the digital age, it is becoming easier for a charismatic leader from an insurgent party to win support. Ten months after it was founded, Podemos, the radical left-wing party in Spain, leads in opinion polls for next year’s general election. The party is led by Pablo Iglesias, a charismatic ponytailed university lecturer who was a familiar face on Spanish television. So far, Podemos’s journey has been even more startling than that of the Five Star Movement in Italy. Less than four years after being founded by the Italian comedian Beppe Grillo, the party came second in the 2013 Italian general election. (It did less well in the 2014 European elections.)

The parties that are surging in Europe are a ragtag bunch. Right-wing populists, in Austria, France, Denmark, Finland, Italy, the Netherlands, Sweden and the UK, have been particularly successful. Separatist parties, in Scotland and Catalonia, are thriving, as are the radical left in Spain (Podemos) and Greece (Syriza). Both oppose austerity and the EU. Unlike Labour and the rest of the mainstream European left, they reject the idea that politicians are restricted in what they can do by globalisation and market economics. Antón Losada, a Spanish ­political scientist, attributes the rise of Podemos to “the lack of response to corruption and institutional failure and the low sensibility shown by the political and economic elites in the face of massive suffering by the Spanish population”. When Iglesias was sworn in as an MEP, he pledged “to recover sovereignty and social rights”.

Ostensibly, little links these disparate groups. The five stars after which the Five Star ­Movement is named – public water, sustainable transport, sustainable ­development, connectivity and environmentalism – would not find much favour with Ukip, the Front National in France or the Danish People’s Party. But what unites the insurgents is often more important than what divides them. They are tapping into people’s fears about globalisation: that jobs are being outsourced and communities are changing beyond recognition. Populists “address cultural identity issues in an era that finds the collective identities of the past in disarray or under threat”, Dick Houtman, an expert on the Party for Freedom in the Netherlands, told me. “People understand immigration and European integration as threats to national identity.”

Voters are rallying against the notion that they have ceded control to faceless bureaucrats or multinational bodies. Populist parties offer voters the prospect, however improbable, of empowerment by ­rejecting the underlying system. They offer bold solutions – leaving the European Union, terminating the Act of Union, abandoning austerity – that leave no room for compromise. Insurgent parties do not accept the need to govern from within a straitjacket determined by the EU, the World Trade Organisation and the International Monetary Fund. They offer voters the promise that their voice will count again. As Mair wrote, “the sense of hostility that some citizens clearly felt towards the political class seemed less important than the indifference with which many more citizens viewed the political world”.

The problem for politicians today is less that they are loathed than that they are seen as irrelevant. The response of many mainstream politicians has been to mimic the populists’ rhetoric, as is happening in Britain because of the rise of Ukip.

Yet a cross in the ballot box for popu-lism is about empowerment more than ­ideology and less an endorsement of a specific policy offer than a rejection of the alternative. Those who blame globalisation for making their lives worse have little enthusiasm for parties that offer only tinkering at the edges. They are voting against a system that they believe has betrayed them.