Never in human history have so many diverse peoples lived together as in our time. Nor has the appeal of democracy ever been so widespread. The promise of equal rights and citizenship held out by modern society has been universally embraced, especially keenly by people long deprived of them. But, as Donald Trump, the favoured candidate of white supremacists, becomes president of the United States, the quintessential multicultural democracy, the long arc of the moral universe, as Martin Luther King called it, does not seem to be bending to justice.

Trump came into political prominence accusing the first black president of the United States of being foreign born; he rose to supreme power stigmatising Mexicans as rapists and Muslims as terrorists. His election victory was engineered by Steve Bannon, the executive chairman of Breitbart News, an online site notorious for its antisemitism, racism, misogyny and xenophobia. The joint arrival of Trump and Bannon in the White House, where they will enjoy nearly unlimited power, completes a comprehensive recent rout of the founding principle of the modern world: that, as the revolutionary phrases of 1776 had it, “all men are created equal”, entitled to the “unalienable rights” of “life, liberty, and the pursuit of happiness”.

Hatemongering against immigrants, minorities and various designated “others” has gone mainstream universally – even in Germany, whose post-Nazi politics and culture were founded on the precept “Never Again”. An era of separatism, in which people barricade themselves in fortresses, united only with those who look and speak like them, has unexpectedly dawned. Back in 1993, the suggestion from Gianfranco Miglio, the intellectual theorist of Italy’s Northern League, that “civilised” Europe should deploy the atavistic nationalism of “barbarian” Europe (the east) as a “frontier guard to block the Muslim invasion” would have seemed preposterous. Today, the demagogues ruling Hungary and Poland claim to be the sentinels of a Christian Europe threatened by Muslim refugees and immigrants. Brexiters in the UK, imitating Tory tactics in London’s mayoral election, conjured up minatory visions of foreigners. A near-majority in the Jewish population of Israel wants the country’s Arab citizens to be expelled. Geert Wilders’ demand for mass deportations of Muslims may help him become prime minister of the Netherlands.

White nationalists in both Europe and America revere Vladimir Putin, who openly rails against “so-called tolerance”, and who inaugurated his regime – and his quest for an “organic” Russian community – with a vicious assault on Chechnya. In India, the world’s largest democracy, Hindu supremacism feeds off a relentless ostracising of minorities. The Turkish president Recep Tayyip Erdoğan is trying to consolidate his support by encouraging attacks on Turkey’s Kurdish citizens. Politicians in Sri Lanka have flourished at the expense of a Tamil minority, which, traumatised by a massacre in 2009, is now routinely victimised by discriminatory policies. Rwanda’s president, Paul Kagame, continues to draw political dividends from his persecution of Hutus. Assaults on religious and ethnic minorities enjoy broad sanction in Pakistan, Bangladesh, Indonesia, Malaysia and Thailand. The much-celebrated advent of democracy in Myanmar now seems to have been a signal for ethnic cleansing.

It was not so long ago that free trade and the “magic of the market”, in the exuberant phrase of the Financial Times commentator Martin Wolf, seemed to be bringing about the benign homogenisation of all human societies. As Louis Vuitton opened in Borneo and the Chinese turned into one of the biggest consumers of French wines, it appeared only a matter of time before free trade and consumer capitalism were followed by the rule of law, the enhanced use of critical reason, the expansion of individual freedom and the tolerance of diversity. Instead, the world at large – from the US to Indonesia – is undergoing a militant tribalisation. The new demagogues combine xenophobia with progressivist rhetoric about decent housing, efficient healthcare systems and better schools. Insisting on linguistic, religious, ethnic, and racial differences, they don’t just threaten free trade, or the globalist dream of achieving cosmopolitan unity through intensified commerce and digital communications. They seem to be deforming nothing less than the secular and egalitarian ideals of modernity.

The emphasis today on cultural difference is unquestionably a response to the painful experience of globalisation

The deformations are particularly ominous in the US, a primarily immigrant country. The abolition of slavery, and an influx of immigrant labour from China, Japan, Ireland, Russia and Germany in the 19th century, turned the US into what Walt Whitman called a “teeming nation of nations”. American politicians and publicists of varying political commitments have since insisted that they are engaged in building a multiracial “city upon a hill”, a country that, dedicated to equal rights and potential for all its citizens, would be an example to all people on earth. Their claims to a quasi-providential mission have been strengthened by the fact that many among the huddled masses around the world, as well as new immigrants in the country, have eagerly wished to be American.

It is also true that the American ideal of the melting pot appears to have little scope for an organic community of the kind Putin, or Marine Le Pen and Nigel Farage invoke. Yet the treacherous fantasy of a homogenised citizenry has repeatedly erupted in the US; and this time it threatens democracy everywhere in the world.

The emphasis today on cultural identity and difference is unquestionably a response to the painful and bewildering experience of globalisation. Those vowing to “take back control” from unaccountable technocracies and opaque financial markets hope to reconstruct a political space by forging afresh the sovereign “people” – a political project that is most quickly achieved by identifying the “enemies” of the people. Ethnic and religious minorities have always been scapegoats for the suffering inflicted by impersonal markets – the word antisemitism was coined in the late 1870s during a severe economic downturn when demagogues channelled mass rage at Jewish populations.

But this explanation has an even more disturbing aspect, which we should not flinch from. The identification and demonisation of racial and ethnic “others” is far from being an aberration in liberal democracy. Nor is it merely a pathology unleashed by economic shocks. Rather, such injustices are central to democracy, as conceived and practised for much of modern history, and they are inseparable from liberal ideals of reason and progress.

***

The African American thinker WEB Du Bois had diagnosed the built-in contradictions of democracy and liberalism as early as the 19th century. In his view slavery had violently coerced Africans into a world economic system, and then global capitalism, binding together more people of different social and historical backgrounds, had piled new economic inequalities on to older racial prejudices and discrimination. Both forms of degradation were vital to the making of prosperous democracies in the Atlantic west; and they made it arduous, if not impossible, for the degraded to realise the modern promise of freedom and equality. “The problem of the 20th century,” Du Bois predicted in 1903, would be “the problem of the colour-line.”

Du Bois wrote as Jim Crow segregation in the industrialising US cancelled gains from the abolition of slavery and as white men scrambling for colonies and empires in Asia and Africa built new racial hierarchies. He would later conclude that the end of slavery in the American south had actually enabled industrial capitalists in the north to expand globally, and, together with their white European counterparts, help entrench “a new industrial slavery of black and brown and yellow workers in Africa and Asia”. Du Bois feared that the “colour of the skin” and “texture of the hair” would become the basis of denying “the opportunities and privileges of modern civilisation” to many.

WEB Du Bois diagnosed the built-in contradictions of democracy and liberalism as early as the 19th century. Photograph: Alamy Stock Photo

The 20th century seemed to both prove and disprove Du Bois’s anxieties about the exclusionary nature of modern politics and economy. Antisemitism, simmering through successive political and economic crises in Europe, exploded in the worst crime in human history. Ruthless imperialists in Asia and Africa presided over wars and famines that killed countless millions. Yet decolonisation led to the creation of independent nation states with egalitarian ideals, followed belatedly by the end of apartheid in South Africa. Some of these globalised economies have appeared in recent years to outpace those of their former western overlords. The problem of the colour line was tackled by the civil rights movement and then seemingly partly solved by a series of successful black politicians, athletes, pop stars, artists and intellectuals; it seemed to have been finally cracked in 2008 when the son of a Kenyan Muslim was elected president of the United States.

But hopes for a post-racial democracy were always extravagant. Obama’s own sanguine attempt at colour-blindness was mocked by, among other things, widely circulating photos that depicted him as a monkey. Ethnic-racial separation has remained starkly evident in the killing or cruel treatment of minorities, housing discrimination against them in major cities and the destitute conditions of many African American and Native American communities. Moreover, reactionary tribalism, or the political urge to create a society of unequal men and women, has never lacked potent sponsors in the US.

By the 1970s rising extreme right groups, the Minutemen, the American Nazi party, the Aryan Nations and a revived Ku Klux Klan were leading a white backlash against the civil rights and feminist movements. The Turner Diaries – a cult 1978 novel by William Pierce, founder of the white nationalist organisation National Alliance – incandescently evokes an America ruled by “swarthy Jewboys” and overrun by African Americans, who have been freed by politically correct legislation to deprive white men of their guns and rape white women.

The destabilisation of the old racial order and gender roles spawned a netherworld of political rage, manifest in recent years in the rise of white militias, attacks on abortion clinics and random shootings. Timothy McVeigh’s murder of 168 Americans in Oklahoma City in 1995 now seems an early salvo in what the isolationist conservative Patrick J Buchanan called a “war for the soul of America”. McVeigh was known to rail against feminism and the political correctness that, in his view, pampered African Americans. Writing about the destruction of the white middle class and the American dream in general in a small-town newspaper in 1992, McVeigh, then a young veteran of the Gulf war, chillingly anticipated our age of anger.

Racism on the rise? You had better believe it. Is this America’s frustrations venting themselves? Is it a valid frustration? Who is to blame for the mess? At a point when the world has seen communism falter as an imperfect system to manage people, democracy seems to be headed down the same road. No one is seeing the “big” picture.

The big picture in 1992 for many, Democratic as well as Republican, was the “end of history” and the worldwide triumph of American-style democracy and capitalism. In this supposedly post-political phase, when national politics seemed merely an adjunct to transnational markets and information networks, it was left to political outliers such as Buchanan to demand social and economic justice for white American workers. Calling for economic protectionism, an end to immigration and toughness with minorities, Buchanan anticipated the xenophobic nationalism of Trump.

***

It was America’s founders, however, who first betrayed the acute tensions in the modern ideologies of individual emancipation. They indeed committed themselves, as Obama asserted in his farewell speech in Chicago this week, to a radical political experiment with their belief in the liberty and equality of every person; but they formulated their self-evident truths in the same Virginian swamp where slaves languished. As it turned out, a mixed and extremely unequal population couldn’t but exacerbate the challenges of realising the universal community of freedom in the US – not to mention in the rest of the world – that immigrants, free traders and imperialists knit closely together.

European settlers, traders and colonists from the 17th century onwards had represented many of the non-European peoples they managed to subdue as uncivilised and inferior, if not candidates for elimination. Racial categories became steadily indispensable to the settlers and colonials of the New World. By the late 18th century, however, people who had been strong-armed into the modern world economy posed a serious moral and political dilemma to those affirming universal human equality and freedom. One way to escape this was to distinguish between those who are properly human and those who aren’t; those who deserve freedom and those who don’t. Thus, a priori distinctions between human and non-human, reason and unreason, civilisation and barbarism underpinned the modern ideals of freedom and democracy from the time they were formulated. John Stuart Mill was upholding these older hierarchies when he, justifying British rule over India, wrote in 1859 that “despotism is a legitimate mode of government in dealing with barbarians”.

Though opposed in principle to slavery, many Enlightenment thinkers and their adepts simply assumed that democratic principles – liberty, equality, toleration, natural rights and human dignity – applied only to civilised white men. Colonised, enslaved and indigenous peoples did not seem capable of reason– the unique characteristic apparently of the human subject liberated from religion and tradition. If David Hume was “apt to suspect the Negroes” to be “naturally inferior to the whites”, Montesquieu had little doubt that they were “barbarians”. Voltaire, who like John Locke held stocks in a company profiting from the slave trade, thought that blacks had only “a few more ideas than animals”. Obama claimed in his speech that African Americans protesting against racial discrimination are demanding “the equal treatment that our founders promised”. In fact, Thomas Jefferson, a slave owner, believed that blacks were “inferior to whites in the endowments of both body and mind” and that his white American compatriots had no choice but to exterminate Native Americans, “ignorant savages” and “beasts”.

Such obsessive dehumanising might seem to negate the humanist ideals that became institutionalised in the American revolution. But, as the Swedish writer Gunnar Myrdal pointed out in his classic study An American Dilemma: The Negro Problem and Modern Democracy (1944), people “placed lower in the biological order than the white man and nearer to the animals” could then be “kept outside the white man’s social and moral order”. Not surprisingly, the pseudo-science of phrenology, which posited biological differences between races, was nowhere more popular than in the US, where white men used it to make the fiction of racial superiority appear a self-evident truth.

“How is it,” Samuel Johnson caustically remarked in 1776, “that we hear the loudest yelps for liberty among the drivers of Negroes?” However, American anti-colonists, devotees of Locke, linked liberty to property rights rather than egalitarian democracy (McVeigh was prone to invoke, not entirely inaptly, both Locke and Jefferson). The slaves’ conspicuous lack of liberty did trouble the conscience of Jefferson and his colleagues, but their ambiguous response was to promote racial separation. A new book by the historian Nicholas Guyatt, Bind Us Apart, argues that “separate but equal” – the notorious reasoning by the US supreme court in 1896 that enshrined Jim Crow segregation in law – is “a founding principle of the United States”. Guyatt, building on the pioneering work of the historian Edmund Morgan, demonstrates that America’s founders were obsessed with the “mental and political compromise” of racial separation long before the American south institutionalised segregation in the wake of the civil war. In fact, American leaders kept toying with the abhorrent (and unworkable) prospect of mass deportations to Africa right up until the civil war.

Racial degradation of non-whites became a form of democratic solidarity – a way to unite white 'wage slaves'

The more inclusive and equal order for white Americans promoted by Andrew Jackson and the Jacksonians in the first half of the 19th century managed to degrade Hispanics, Native Americans, slaves, free black people and women. More such paradoxes came to define an increasingly vibrant American democracy after the emancipation of slaves and the end of the civil war. It was then that racial separation and exclusion came to unmistakably demarcate the community deserving of freedom and equality in the US. Segregated schools, railroad cars and lunch counters across the American south would for decades bolster the fiction that the races are separate but equal, while the lynching and disfranchisement of black people underscored which of the races was on top.

In 1882, Congress passed the Chinese Exclusionary Act, which prohibited the entry of Chinese labourers. The xenophobic legislation, which inspired antisemitic demagogues as far as Vienna, made no sense: Chinese immigrants constituted a mere 0.002% of the US population in 1880. But this was also the time when a country previously abundant in jobs and land was discovering the trauma of unemployment amid economic crisis and social conflict; and many citizens came to cherish their citizenship as an exclusive privilege that should not be made available to all and sundry, especially their racial underlings.

Racial degradation of non-whites became a form of democratic solidarity in the US in the turbulent late 19th century. For both rightwing and leftwing populists, it was a way to unite white “wage slaves” against Asian immigrants and African Americans, and heal the wounds to their dignity. Fresh immigrants from Ireland could also achieve honorary whiteness by persecuting African Americans – the colour line was negotiable for some people at least. If antisemitism in Europe was the socialism of fools, racism in late 19th-century America was the democracy of the aggrieved left-behinds and pushy newcomers.

Many progressives, as Du Bois saw clearly, were complicit in it. The trust-busting American president Theodore Roosevelt swore by political equality, economic security and social opportunity for all Americans. But his inclusive order pitilessly rejected non-whites. Wishing to “tighten”, in Henry James’s mordant assessment, “the screws of the national consciousness as they have never been tightened before”, Roosevelt hoped that war and conquest abroad would forge racial unity and democracy at home. The original liberal internationalist Woodrow Wilson was hardly atypical in his reverence for what he called the “great Ku Klux Klan”, which had emerged after the end of slavery to protect whites from “the votes of ignorant Negroes”.

A widespread faith in eugenics, the much revered pseudo-science of the early 20th century, went on to shape the 1924 Immigration Act and its system of quotas, which favoured newcomers from north-western Europe over racially suspect (often Jewish) south-eastern Europeans and excluded “descendants of slave immigrants” as well as Asian immigrants. More than 100,000 Japanese Americans were incarcerated as the US fought the second world war, ostensibly for freedom, with a segregated army. The civil rights revolution of the 1960s finally ended the terrors of Jim Crow. But increased political clout by African Americans, “the unmeltable ethnics”, in the infamous phrase of the conservative writer Michael Novak, provoked a backlash whose political reverberations can be felt to this day.

The civil rights movement made it impossible to appeal to racial furies as thunderously as George Wallace, the governor of Alabama whose war cry in 1963 was “Segregation now, segregation tomorrow, segregation forever”. It could not, however, prevent white politicians from dog-whistling. Richard Nixon’s Wallace-lite overtures to the “silent majority” evidently aghast at assertive blacks and multiculturalist liberals were refined by Ronald Reagan. As he attacked affirmative action and other gains of the civil rights movement under the guise of promoting such liberal-left causes as “colour blindness” and a “level playing field”, Reagan reached out to white working class voters with code words such as “states’ rights”, “welfare moms”, “quotas” and “reverse racism”. If the Willie Horton ad used in the George HW Bush presidential campaign in 1988 clarified the persisting power of racial dreads, the George W Bush administration’s response to Hurricane Katrina in 2005 suggested that “segregation forever” had long been more than a nasty slogan in large parts of the US. Trump eventually reaped the electoral harvest of a reflexive loathing among many Americans for Obama – “a guy”, as Fox News’ Glenn Beck put it, with “a deep-seated hatred for white people”.

***

The never fully repressed denunciation on the basis of race has exultantly returned in our own time. Trump’s consigliere, Steve Bannon, responded to the recent spate of murders by police of unarmed African Americans with: “What if the people getting shot by the cops did things to deserve it? There are, after all, in this world, some people who are naturally aggressive and violent.”

Blithely attacking minorities (and recklessly baiting China), Trump and his confederates have violated even the fragile moratorium on antisemitism in place since the exposure of Nazi crimes. Their abrupt legitimation of vile stereotypes that were supposedly laid to rest ages ago has grim repercussions for the rest of the world’s hybrid and unequal societies. It is in the US that a faith in inevitable and irreversible progress – a “more perfect union” – has long bridged the abyss between the high-minded ideals of democracy and the cruel facts of structural violence and inequality. Obama again insisted this week that America “has been defined by forward motion, a constant widening of our founding creed to embrace all, and not just some”.

A restaurant in Maryland in 1948, with separate entrances for white and black customers Photograph: Joe Schwartz Photo Archive/Corbis via Getty Images

Trump’s ascent reveals the longterm winners and losers of this distinctively American ideology – the most powerful religion of our time. Many white American liberals are perplexed why their reverent invocation of America’s inclusive ideals and their complaints about the identity politics of minorities are met with angry calls to “check your privilege”. But, contrary to many hopes and claims, America’s liberal-democratic order has been largely inclusive for those who are privileged enough to be included in it. Exclusion has become steadily less crudely racial than it was during the unconscionably long era of segregation; but it is determined today by gender, property, educational and economic opportunity as well as by race.

This multidimensional inequality has grown more intolerable during a prolonged economic crisis. It has boosted the appeal of the ethno-racial nationalism that surged in Europe and America during the first phase of intensive globalisation in late 19th century, when, as Du Bois wrote, an American and European elite built up “concentrated economic power and profit greater than the world had envisioned”. Many among the middle and the working classes today feel excluded from both the benefits of the welfare state and the bonanzas of the rich. They aim their rage at both an aloof technocracy and people they suspect of exploiting the taxpayers’ generosity. As in fin de siècle America and Europe, political opportunists try to capitalise on their fears by demonising foreigners, immigrants, refugees – all supposed parasites on the hard-working and cruelly neglected classes that should be weeded out.

We are discovering yet again that an atomised people repoliticises and reconstitutes itself by learning, in the bleak formulation of the Nazi jurist Carl Schmitt, “how to keep at bay something foreign and unequal that threatens its homogeneity”. We are finding out that racism is not simply a product of ignorance, prejudice or arrogance; it endures, despite all our cautionary tales and resolves of “never again”, because its promise of social solidarity serves to assuage human fears and nurture hopes for the future. Racial exclusion, a response to the insatiable modern demand for equality, liberty and dignity, is bound up insidiously with the most virtuous ideals of liberalism and democracy.

This is especially true of the US, which, as Guyatt warns, obviously “bears the scars of its segregated past” but “also retains an instinct for racial separation that manifests itself even among those who forswear racist beliefs”. Bill Clinton surpassed Nixon, Reagan and the Bushes in condemning African Americans to mass incarceration and poverty while deregulating financial markets for the benefit of his patrons on Wall Street. The rhetoric and actions of Trump’s cabinet, the wealthiest and most fanatical yet, will no doubt clarify further the inhuman practices that drive a politics and economy ostensibly devoted to human freedom.

Genuine democratic equality under the Trump administration will be a more formidable challenge than ever before

Those who oppose them should welcome this clarity. It has taken too long for the ellipses, omissions and subterfuges in the American – and now universal – promise of liberty to be widely noticed. Several generations of anti-imperialist thinkers and activists, who intimately experienced the worldwide “industrial slavery” that Du Bois wrote about, repeatedly pointed out that those who promise equal rights universally enforce at the same time a global hierarchy in which those rights are reserved for some and forbidden to others. Certainly, Gandhi would have found very familiar the politicians who guarantee liberty, equality and dignity to people who look like them while flagrantly denying them to those who don’t.

Gandhi would also have recognised, just as his American disciple Martin Luther King did, the imperative of building a civic democracy that takes into account the pluralistic nature of contemporary societies and the apparent incompatibility of competing claims and values: a democracy that acknowledges incommensurate goals and stimulates cooperation and reciprocity rather than competition and animosity between its individual members. The arduous task of creating unity in diversity, among people riven by race, class and gender, never confronted the founders of the United States. Too many complex issues – such as the nature of human freedom and equality – seemed self-evident to them; and too many of their successors also concealed the self-evident contradictions in the American programme by banishing from sight the enslaved, colonised and dispossessed people whose resources and labour enabled the enjoyment of life, liberty and happiness.

Today, as white supremacists prepare to occupy the house built by slaves in Washington DC, it may be hard to resist the fear that these pugnacious men, “struggling to hold on to what they have stolen from their captives”, as James Baldwin put it in 1967, “and unable to look into their mirror, will precipitate a chaos throughout the world that, if it does not bring life on this planet to an end, will bring about a racial war such as the world has never seen”. Certainly, genuine democratic equality under the Trump administration will be a more formidable challenge than ever before. But at least it won’t appear veiled by the illusions of the past – which may give present and future generations a better chance of bending the intractable arc of the moral universe to justice.

• Age of Anger by Pankaj Mishra is published later this month by Allen Lane.