The election of Donald Trump as president of the United States is the biggest political earthquake of our times, and its reverberations are inescapably global. It has fully revealed an enormous pent-up anger – which had first become visible in the mass acclaim in Russia and Turkey for pitiless despots and the electoral triumph of bloody strongmen in India and the Philippines.

The insurgencies of our time, including Brexit and the rise of the European far right, have many local causes – but it is not an accident that demagoguery appears to be rising around the world. Savage violence has erupted in recent years across a broad swath of territory: wars in Ukraine and the Middle East, insurgencies from Yemen to Thailand, terrorism and counter-terrorism, economic and cyberwar. The conflicts, not confined to fixed battlefields, feel endemic and uncontrollable. Hate-mongering against immigrants and minorities has gone mainstream; figures foaming at the mouth with loathing and malice are ubiquitous on old and new media alike.

There is much dispute about the causes of this global disorder. Many observers have characterised it as a backlash against an out-of-touch establishment, explaining Trump’s victory – in the words of Thomas Piketty – as “primarily due to the explosion in economic and geographic inequality in the United States”. Liberals tend to blame the racial resentments of poor white Americans, which were apparently aggravated during Barack Obama’s tenure. But many rich men and women – and even a small number of African-Americans and Latinos – also voted for a compulsive groper and white supremacist.

The Nobel prize-winning economist Paul Krugman admitted on the night of Trump’s victory that “people like me – and probably like most readers of the New York Times – truly didn’t understand the country we live in”. Since the twin shocks of Brexit and the US election, we have argued ineffectually about their causes, while watching aghast as the new representatives of the downtrodden and the “left-behind” – Trump and Nigel Farage, posing in a gold-plated lift – strut across a bewilderingly expanded theatre of political absurdism.

But we cannot understand this crisis because our dominant intellectual concepts and categories seem unable to process an explosion of uncontrolled forces.

In the hopeful years that followed the fall of the Berlin Wall in 1989, the universal triumph of liberal capitalism and democracy seemed assured; free markets and human rights would spread around the world and lift billions from poverty and oppression. In many ways, this dream has come true: we live in a vast, homogenous global market, which is more literate, interconnected and prosperous than at any other time in history.

And yet we find ourselves in an age of anger, with authoritarian leaders manipulating the cynicism and discontent of furious majorities. What used to be called “Muslim rage”, and identified with mobs of brown-skinned men with bushy beards, is suddenly manifest globally, among saffron-robed Buddhist ethnic-cleansers in Myanmar, as well as blond white nationalists in Germany. Violent hate crimes have blighted even the oldest of parliamentary democracies, with the murder of the MP Jo Cox by a British neo-Nazi during the venomous campaign for Brexit. Suddenly, as the liberal thinker Michael Ignatieff recently wrote: “Enlightenment humanism and rationalism” can no longer adequately “explain the world we’re living in.”

The largely Anglo-American intellectual assumptions forged by the cold war and its jubilant aftermath are an unreliable guide to today’s chaos – and so we must turn to the ideas of an earlier era of volatility. It is a moment for thinkers such as Sigmund Freud, who warned in 1915 that the “primitive, savage and evil impulses of mankind have not vanished in any individual”, but are simply waiting for the opportunity to show themselves again. Certainly, the current conflagration has brought to the surface what Friedrich Nietzsche called “ressentiment” – “a whole tremulous realm of subterranean revenge, inexhaustible and insatiable in outbursts.”

By contrast, the fundamental premise of our existing intellectual frameworks is the assumption that humans are essentially rational and motivated by the pursuit of their own interests; that they principally act to maximise personal happiness, rather than on the basis of fear, envy or resentment.

The bestseller Freakonomics is a perfect text of our time in its belief that “incentives are the cornerstone of modern life,” and “the key to solving just about any riddle”. From this view, the current crisis is an irruption of the irrational – and confusion and bewilderment are widespread among political, business and media elites. The ordinarily stolid Economist has lately lurched from dubious indignation over “post-Truth politics” to the Rip Van Winkle-ish declaration of “The New Nationalism”. Many other mainstream periodicals now read like parodies of New Left Review, as they attend belatedly to the failings of global capitalism – most egregiously, its failure to fulfil its own promise of general prosperity.

We can now see, all too clearly, a widening abyss of race, class and education in Britain and the US. But as explanations proliferate, how it might be bridged is more unclear than ever. Well-worn pairs of rhetorical opposites, often corresponding to the bitter divisions in our societies, have once again been put to work: progressive v reactionary, open v closed, liberalism v fascism, rational v irrational. But as a polarised intellectual industry plays catch-up with fast-moving events that it completely failed to anticipate, it is hard to avoid the suspicion that our search for rational political explanations for the current disorder is doomed. All of the opponents of the new “irrationalism” – whether left, centre, or right – are united by the presumption that individuals are rational actors, motivated by material self-interest, enraged when their desires are thwarted, and, therefore, likely to be appeased by their fulfilment.

This notion of human motivation deepened during the Enlightenment, whose leading thinkers, despising tradition and religion, sought to replace them with the human capacity to rationally identify individual and collective interests. The dream of the late 18th century, to rebuild the world along secular and rational lines, was further elaborated in the 19th century by the utilitarian theorists of the greatest happiness for the greatest number of people – and this notion of progress was embraced by socialists and capitalists alike.

After the collapse of the socialist alternative in 1989, this utopian vision took the form of a global market economy dedicated to endless growth and consumption – to which there would be no alternative. According to this worldview, the dominance of which is now nearly absolute, the human norm is Homo economicus, a calculating subject whose natural desires and instincts are shaped by their ultimate motivation: to pursue happiness and avoid pain.

This simple view always neglected many factors ever-present in human lives: the fear, for instance, of losing honour, dignity and status, the distrust of change, the appeal of stability and familiarity. There was no place in it for more complex drives: vanity, fear of appearing vulnerable, the need to save face. Obsessed with material progress, the hyperrationalists ignored the lure of resentment for the left-behind, and the tenacious pleasures of victimhood.

And yet modern history provides enormous evidence for the persistent power of unreason. It was not so long ago – in the early 19th century – that French pretensions to a rational, universal, and cosmopolitan civilisation first provoked resentful Germans into the militant expression of what we now call “cultural nationalism”: the assertion of authentic culture rooted in national or regional character and history.

One revolution after another since then has demonstrated that feelings and moods change the world by turning into potent political forces. Fear, anxiety and a sense of humiliation were the principal motive of Germany’s expansionist policy in the early 20th century – and it is impossible to understand the current upsurge of anti-western sentiment in China, Russia and India without acknowledging the role played by humiliation.

Yet a mechanistic and materialist way of conceiving human actions has become entrenched, in part because economics has become the predominant means of understanding the world. A view that took shape in the 19th century – that there is “no other nexus between man and man than naked self-interest” – has become orthodoxy once again in an intellectual climate that views the market as the ideal form of human interaction and venerates technological progress and the growth of GDP. All of this is part of the rigid contemporary belief that what counts is only what can be counted and that what cannot be counted – subjective emotions – therefore does not.

A Brexit supporter, and a Vote Remain campaigner exchange views in Market Square, Northampton, on 31 May Photograph: Christopher Furlong/Getty Images

Our current disregard of non-economic motivations is even more surprising when we learn that less than a century ago, the Enlightenment’s “narrow rational programme” for individual happiness had already become “the butt of ridicule and contempt” – as the Austrian modernist writer Robert Musil observed in 1922. Indeed, the pioneering works of sociology and psychology as well as modernist art and literature of the early 20th century were defined in part by their insistence that there is more to human beings than rational egoism, competition and acquisition, more to society than a contract between logically calculating and autonomous individuals, and more to politics than impersonal technocrats devising hyper-rational schemes of progress with the help of polls, surveys, statistics, mathematical models and technology.

Writing in the 1860s, during the high noon of 19th-century liberalism, Fyodor Dostoevsky was one of the first modern thinkers to air the suspicion, now troubling us again, that rational thinking does not decisively influence human behaviour. He pitted his Underground Man – the quintessential loser dreaming of revenge against society’s winners – against the idea of rational egoism, or material self-interest, then popular in Russia among eager readers of John Stuart Mill and Jeremy Bentham. Dostoevsky’s protagonist obsessively assaults the shared rationalist assumptions of both capitalists and socialists: that human beings are logically calculating animals, driven by perceived incentives:

Oh, tell me who was it first announced, who was it first proclaimed, that man only does nasty things because he does not know his own interests; and that if he were enlightened, if his eyes were opened to his real normal interests, man would at once cease to do nasty things, would at once become good and noble because, being enlightened and understanding his real advantage, he would see his own advantage in the good and nothing else?

Dostoevsky defined a style of thought that was later elaborated by Nietzsche, Freud, Max Weber and others – who mounted a full-blown intellectual revolt against the oppressive certainties of rationalist ideologies, whether left, right or centre. This is an intellectual revolution that is barely remembered today – but it erupted at an emotional and political moment that would seem eerily familiar to us: a period of uneven and disruptive economic growth, distrust of politicians, fear of change, and anxiety about rootless cosmopolitans, aliens and immigrants.

This was an era when the disaffected masses – recoiling from the 19th century’s prolonged experiment in laissez-faire economic rationalism – had begun to fall for radical alternatives, in the form of blood-and-soil nationalism and anarchist terrorism. This anti-liberal political uprising forced many of those we now regard as central figures of 20th-century intellectual life to question their fundamental notions of human behaviour, and to discard the positivist nostrums that had taken root in the previous century.

By the late 1850s, Charles Darwin had already shattered the notion that human beings could control how they develop – let alone build a rational society. Novelists, sociologists and psychologists examining the turbulent mass societies of the late 19th century concluded that human actions could not be reduced to single causes, whether religious and ideological faith, or the rationality of self-interest.

Freud, who lived in turn-of-the-century Vienna while demagogues were scapegoating Jews and liberals for the mass suffering inflicted by industrial capitalism, came to see the rational intellect as “a feeble and dependent thing, a plaything and tool of our impulses and emotions”. “One gets the impression,” Freud wrote in The Future of an Illusion (1927) “that culture is something imposed on a reluctant majority by a minority that managed to gain possession of the instruments of power and coercion.” Long before the 20th century’s explosions of demagoguery, Max Weber, as he observed Germany’s hectic industrialisation, presciently speculated that individuals, unmoored by socioeconomic turmoil and alienated by bureaucratic rationalisation, could become vulnerable to a despotic leader.

The problem for these critics of Enlightenment rationalism, as Robert Musil defined it, was not that we “have too much intellect and too little soul”, but that we have “too little intellect in matters of the soul”. We suffer even more from this problem today as we struggle to make sense of the outbreaks of political irrationalism. Committed to seeing the individual self as a rational actor, we fail to see that it is a deeply unstable entity, constantly shaped and reshaped in its interplay with shifting social and cultural conditions. In our own time, amid what Hannah Arendt described as a “tremendous increase in mutual hatred and a somewhat universal irritability of everybody against everybody else”, this fragile self has become particularly vulnerable to ressentiment.

Ressentiment – caused by an intense mix of envy, humiliation and powerlessness – is not simply the French word for resentment. Its meaning was shaped in a particular cultural and social context: the rise of a secular and meritocratic society in the 18th century. Even though he never used the word, the first thinker to identify how ressentiment would emerge from modern ideals of an egalitarian and commercial society was Jean-Jacques Rousseau. An outsider to the Parisian elite of his time, who struggled with envy, fascination, revulsion and rejection, Rousseau saw how people in a society driven by individual self-interest come to live for the satisfaction of their vanity – the desire and need to secure recognition from others, to be esteemed by them as much as one esteems oneself.

But this vanity, luridly exemplified today by Donald Trump’s Twitter account, often ends up nourishing in the soul a dislike of one’s own self while stoking impotent hatred of others; and it can quickly degenerate into an aggressive drive, whereby individuals feel acknowledged only by being preferred over others, and by rejoicing in their abjection. (As Gore Vidal pithily put it: “It is not enough to succeed. Others must fail.”)

Such ressentiment breeds in proportion to the spread of the principles of equality and individualism. In the early 20th century, the German sociologist Max Scheler developed a systematic theory of ressentiment as a distinctly modern phenomenon – ingrained in all societies where formal social equality between individuals coexists with massive differences in power, education, status, and property ownership. In an era of globalised commerce, these disparities now exist everywhere, along with enlarged notions of individual aspiration and equality. Accordingly, ressentiment, an existential resentment of others, is poisoning civil society and undermining political liberty everywhere.

But what makes ressentiment particularly malign today is a growing contradiction. The ideals of modern democracy – the equality of social conditions and individual empowerment – have never been more popular. But they have become more and more difficult, if not impossible, to actually realise in the grotesquely unequal societies created by our brand of globalised capitalism.

The past two decades of hectic globalisation have brought us closer than ever before to the liberal Enlightenment ideal of a universal commercial society of self-interested, rational and autonomous individuals – one that was originally advocated in the 18th century by such thinkers as Montesquieu, Voltaire, Adam Smith, and Kant. In the 19th century, it was still possible for Marx to sneer at Jeremy Bentham for assuming “the modern shopkeeper, especially the English shopkeeper, as the normal man”. In our own time, however, the ideology of neoliberalism – a market-centric hybrid of Enlightenment rationalism and 19th-century utilitarianism – has achieved near total domination in the economic and political realm alike.

The success of this universal creed can be attested by many innovations of recent decades that now look perfectly natural. The rational market is expected to ensure the supply of valuable products and services, while the task of governments is to ensure fair competition, which produces “winners” and “losers”. The broad intellectual revolution in which an all-knowing market judges failure and success has even more forcefully insisted on the rationality of the individual.

Issues of social justice and equality have receded along with conceptions of society or community – to be replaced by the freely choosing individual in the marketplace. According to the prevailing view today, the injustices entrenched by history or social circumstances cease to matter: the slumdog, too, can be a millionaire, and the individual’s failure to escape the underclass is self-evident proof of his poor choices.

But this abstract conception has no room for the emotional situation of real, flesh-and-blood people – and how they might act within concrete social and historical settings.

One of the first people to notice the disturbing complex of emotions we now see among self-seeking individuals around the world was Alexis de Tocqueville – who was already worried in the 1830s that the American promise of meritocracy, its uniformity of culture and manners, and “equality of conditions” would make for immoderate ambition, corrosive envy and chronic dissatisfaction. The passion for equality, he warned, could swell “to the height of fury” and lead many to acquiesce in a curtailment of their liberties, and to long for the rule of a strongman.

As De Tocqueville pointed out, people liberated from old hierarchies “want equality in freedom, and, if they cannot get it, they still want it in slavery.”

We witness a universal frenzy of fear and loathing today because the democratic revolution De Tocqueville witnessed has spread from its American centre to the remotest corners of the world. The rage for equality is conjoined with the pursuit of prosperity mandated by the global consumer economy, aggravating tensions and contradictions in inner lives that are then played out in the public sphere.

“To live in freedom,” De Tocqueville warned, “one must grow used to a life full of agitation, change and danger.” This kind of life is barren of stability, security, identity and honour, even when it overflows with material goods. Nevertheless, it is now commonplace among people around the world that rational considerations of utility and profit – the needs of supply chains and the imperatives of quarterly shareholder returns – uproot, humiliate and render obsolete.

The widespread experience of the maelstrom of modernity has only heightened the lure of ressentiment. Many new individuals now “live in freedom”, in De Tocqueville’s words, even as they are enslaved by finely integrated political, economic and cultural powers: the opaque workings of finance capital, the harsh machinery of social security, juridical and penal systems, and the unrelenting ideological influence of the media and the internet.

Never have so many free individuals felt so helpless – so desperate to take back control from anyone they can blame for their feeling of having lost it. It should not be surprising that we have seen an exponential rise in hatred of minorities, the main pathology induced by political and economic shocks. These apparent racists and misogynists have clearly suffered silently for a long time from what Albert Camus called “an autointoxication – the evil secretion, in a sealed vessel, of prolonged impotence”. It was this gangrenous ressentiment, festering for so long in places such as the Daily Mail and Fox News, that erupted volcanically with Trump’s victory.

Rich and poor alike voting for a serial liar and tax dodger have confirmed yet again that human desires operate independently of the logic of self-interest – and may even be destructive of it. Our political and intellectual elites midwifed the new “irrationalism” through a studied indifference to the emotional dislocation and economic suffering induced by modern capitalism. Not surprisingly, they are now unable to explain its rise. Indeed, their universal assumption, hardened since 1989, that there are no alternatives to western-style democracy and capitalism – the famous “end of history” – is precisely what has made us incapable of grasping the political phenomena shaking the world today.

It is clear now that the exaltation of individual will as something free of social and historical pressures, and as flexible as markets, concealed a breathtaking innocence about structural inequality and the psychic damage it causes. The contemporary obsession with individual choice and human agency disregarded even the basic discoveries of late-19th-century sociology: that in any mass society, life chances are unevenly distributed, there are permanent winners and losers, a minority dominates the majority, and the elites are prone to manipulate and deceive.

Even the terrorist attacks of 9/11 left undisturbed the vision in which a global economy built around free markets, competition and rational individual choices would alleviate ethnic and religious differences and usher in worldwide prosperity and peace. In this utopia, any irrational obstacles to the spread of liberal modernity – such as Islamic fundamentalism – would be eventually eradicated. Fantasies of a classless and post-racial society of empowered rational-choice actors bloomed as late as 2008, the year of the most devastating economic crisis since the Great Depression.

Today, however, the basic assumptions of cold war liberalism lie in ruins – after decades of intellectual exertion to construct flimsy oppositions between the rational west and the irrational east. The political big bang of our time does not merely threaten the vanity projects of an intellectual elite, but the health of democracy itself – the defining project of the modern world. Since the late 18th century, tradition and religion have been steadily discarded, in the hope that rational, self-interested individuals can form a liberal political community that defines its shared laws, ensuring dignity and equal rights for each citizen, irrespective of ethnicity, race, religion and gender. This basic premise of secular modernity, which earlier only seemed menaced by religious fundamentalists, is now endangered by elected demagogues in its very heartlands, Europe and the US.

Where do we go from here? We can of course continue to define the crisis of democracy through reassuring dualisms: liberalism v authoritarianism, Islam v modernity, and that sort of thing. It may be more fruitful to think of democracy as a profoundly fraught emotional and social condition – one which, aggravated by turbo-capitalism, has now become unstable. This might allow us to examine the workings of ressentiment across varied countries and classes, and to understand why ethno-nationalist supremacy has grown alongside economic stagnation in America and Britain, even as it flourishes alongside economic expansion in India and Turkey. Or, why Donald Trump, the flashy plutocrat tormented by his lowly status among Manhattan’s cultivated liberals, obsessively baits the New York Times and calls for a boycott of the Broadway show Hamilton.

* * *

That a rancorous Twitter troll will soon become the world’s most powerful man is the latest of many reminders that the idealised claims of western elites about democracy and liberalism never actually conformed to the political and economic reality at home. A rowdy public culture of disparagement and admonition does not hide the fact that the chasm of sensibility between a technocratic elite and the masses has grown. Everywhere, a majority that was promised growing equality sees social power monopolised by people with money, property, connections and talent; they feel shut out from both higher culture and decision-making.

Many people find it easy to aim their rage against an allegedly cosmopolitan and rootless cultural elite. Objects of hatred are needed more than ever during times of crisis, and rich “citizens of nowhere” – as Theresa May dubbed them – conveniently embody the vices of a desperately sought-after but infuriatingly unattainable modernity. And so globalisation, which promotes integration among shrewd elites, helps incite ressentiment everywhere else, especially among people forced against their will into universal competition.

In search of a balm for these wounds, many intellectuals have embraced nostalgic fantasies of vanished unity. Earlier this year, the New York Times columnist David Brooks returned from communist Cuba gushing about Cubans’ “fierce love of country, a sense of national solidarity and a confident patriotic spirit that is today lacking in the United States.” More recently, Simon Jenkins, in this newspaper, and the intellectual historian Mark Lilla – in a widely circulated New York Times opinion piece – have urged the rejection of “identity liberalism” and the necessity of embracing national unity and common identity. As Trump’s victory was declared, Simon Schama tweeted that we need a new Churchill to save democracy in Europe and America.

A Trump supporter in Ambridge, Pennsylvania during the presidential election campaign. Photograph: Evan Vucci/AP

Such breast-beating amounts to a truly irrational demand: that the present abolish itself, making way for a return to the past. Ideally, to the time when paternalistic white liberals occupied the vital centre, little disturbed by the needs and desires of history’s forgotten, humiliated and silenced people.

These lamentations for simpler times – that all we lack is the right sort of spine-stiffening democratic leader, or rational culture, or cultural unity, or patriotic spirit – ignore the fragmented nature of our politics. Social and technological developments are not liberal or conservative, democratic or authoritarian; they are as prone to enshrine LGBT rights as to reinstate torture and disseminate fake news. Nor does the longing for the good old days adequately respond to the massive crisis of legitimacy facing democratic institutions today.

Political antidotes to the sinister pathologies unleashed by Putin, Erdoğan, Modi, Brexit and Trump require a reckoning with the bad new days – something a lot more forward-looking than models of solidarity inspired by Cuba or Churchill, nationalist pedagogies for the oppressed, or dauntless faith in globalisation eventually delivering the promised goods.

This work is necessary – but it can only proceed with a more sophisticated analysis of how today’s landscape of hyperrational power has coerced a new and increasingly potent irrationalism into existence. And such analyses would require, above all, a richer and more varied picture of human experience and needs than the prevailing image of Homo economicus. This intellectual effort – which was first undertaken more than a century ago by the thinkers cited here – would necessarily take us beyond liberalism and its faith in the curative power of economic growth.

What Robert Musil called the “liberal scraps of an unfounded faith in reason and progress” have yet again failed modern human beings in their all-important task of understanding their experience. We once more confront the possibility, outlined in Musil’s great novel about the collapse of liberal values, The Man Without Qualities, that the characteristic desolation of the modern human being – his “immense loneliness in a desert of detail, his restlessness, malice, incomparable callousness, his greed for money, his coldness and violence’ – is “the result of the losses that logically precise thinking has inflicted on the soul”.

For nearly three decades, the religion of technology and GDP and the crude 19th-century calculus of self-interest have dominated politics and intellectual life. Today, the society of entrepreneurial individuals competing in the rational market reveals unplumbed depths of misery and despair; it spawns a nihilistic rebellion against order itself.

With so many of our landmarks in ruins, we can barely see where we are headed, let alone chart a path. But even to get our basic bearings we need, above all, greater precision in matters of the soul. The stunning events of our age of anger, and our perplexity before them, make it imperative that we anchor thought in the sphere of emotions; these upheavals demand nothing less than a radically enlarged understanding of what it means for human beings to pursue the contradictory ideals of freedom, equality and prosperity.

Otherwise, in our sterile infatuation with rational motivations and outcomes, we risk resembling those helpless navigators who, De Tocqueville wrote, “stare obstinately at some ruins that can still be seen on the shore we have left, even as the current pulls us along and drags us backward toward the abyss”.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.