The rise of populism across western Europe and the US – especially in its radical right form – poses more fundamental questions for democrats than has been acknowledged. Whether we are talking about Ukip, Beppe Grillo's Five Star Movement or the Tea Party, populists of all kinds are exposing old and hidden fault lines in democracy, and mainstream democrats need a greater alertness to the nature of the threat. Modern democracy, like a hot-air balloon untethered from the ground, is suddenly floating free and its destination is not yet known.

Populists pose a basic question: why is democracy not run as the true expression of a morally pure "will of the people" against a self-serving and corrupt political, bureaucratic, plutocratic or legal elite? This is a forceful question as old as democracy itself and it reveals what has become liberal democracy's unspoken compromise – democracy is bounded by institutions, laws and constitutional limits. It is democracy through pluralism and compromise; "minorities rule" as the American democratic theorist, Robert Dahl, described it. For populists, the problem with this notion is that they have their eyes on what they perceive as the majority (it usually isn't, in fact) against constitutional, legal or international constraints that have been placed on the "general will".

Mainstream democrats take their cue from American republican democracy with its checks and balances and self-restraint. This is an impediment to the true democracy for populists. They wish to sweep away any barrier to their desired ends – whether of the left or the right.

So the Tea Party proposes a radical reduction of the role of the federal government in the US political system. The FPÖ challenged the authority of Austrian courts with respect to upholding minority rights. Ukip demands a UK withdrawal from the EU. The Front National drives an anti-Islamic and anti-Gypsy agenda in France. Geert Wilders' PVV – following in the footsteps of Pim Fortuyn – also confronts fears over the growth of Islam and its purported incompatibility with Dutch values. Venezuela's Hugo Chávez created a parallel state and augmented his own constitutional power. Viktor Orbán's Fidesz rewrote the Hungarian constitution to give the executive more authority over the courts and to safeguard "traditional family values".

The binding element to all of these movements and parties is that they are not simply seeking to compete for ideas, policies and power but want to change the rules of the democratic game in favour of executive, majority rule. They are democrats but majority rule is their guiding force rather than a legally enshrined pluralism with minority protection. As my new Policy Network report, Democratic Stress, the Populist Signal and Extremist Threat shows, the upshot is modern (liberal) democracy in a state of stress.

Underlying the growth of these populist movements is a series of stressors that come to bear on liberal democracy and its mainstream party systems. They are socioeconomic, cultural and political in nature.

As the economy has moved away from mass production and many have lost out, socioeconomic change has loosened the ties of parties. Austerity looms large but is one factor among many. The rise of a politics of plural cultural identities catalysed by modern technology, transport, communications and media has further loosened the grip of mainstream parties on a solid and predictable base. Finally, political changes such as the expansion of the EU's acquis communautaire and the increasing comfort of mainstream parties within the system has created an opening for political challenger brands. Those challenger brands are the populists – more so than the green movement and even nationalists.

Mainstream parties have to prove that republican, pluralistic democracy, despite its frustrations and complexities, can be navigated through the trade-offs that all societies face better than populism can. More often than not, populists have simplistic and, in the worst case, highly damaging policies which if ever enacted, could cause significant harm. The desire for a return to economic growth, a sustainable welfare state or reducing debts is just as great as controlling immigration more tightly or seizing significant powers back from the EU. Mainstream parties can cope with these trade-offs between these demands and needs better than populists can – as long as they craft a viable statecraft.

Just maybe there is some truth in the populist critique of political elites – in Brussels, Washington and right through western democracies – and the way they have embedded their own self-interest in the system. Mainstream parties have lost their edge. They have grown comfortable, closed and politically nepotistic – relying on voters having nowhere else to go. That works for a while but becomes progressively more difficult to sustain. Mainstream democracy needs to become a contact sport again – with greater openness and engagement between the people and those who seek to represent them. Parties need to open up to real change and diversity.

Populism doesn't have the answers – you don't need to be Thomas Jefferson, James Madison or even Edmund Burke to see that. In many ways, however, it is posing some of the right questions. New policy approaches are necessary but not sufficient. The political mainstream needs to ask more fundamental questions of itself and its ability to govern with real legitimacy. That is if they are not to continue to age as the societies around them age. People will be left with a choice between a tired political mainstream elite and populists with all of the answers but few solutions. It's not a choice many will savour.