The British empire, George Orwell wrote, was "despotism with theft as its final object". So what has made imperialism an intellectual fashion in our own time, reopening hoary disputes about whether it was good or bad? After five years as a colonial policeman in Burma, where he found himself shooting an elephant to affirm the white man's right to rule, Orwell was convinced that the imperial relationship was that of "slave and master". Was the master good or bad? "Let us simply say," Orwell wrote, "that this control is despotic and, to put it plainly, self-interested." And "if Burma derives some incidental benefit from the English, she must pay dearly for it."

Orwell's hard-won insights were commonplace truisms for millions of Asians and Africans struggling to end western control of their lands. Their descendants can only be bewildered by the righteous nostalgia for imperialism that has recently seized many prominent Anglo-American politicians and opinion-makers, who continue to see Asia through the narrow perspective of western interests, leaving unexamined and unimagined the collective experiences of Asian peoples.

Certainly, as Joseph Conrad wrote in 1902, "the conquest of the earth, which mostly means the taking it away from those who have a different complexion or slightly flatter noses than ourselves, is not a pretty thing when you look into it too much." Two years after Conrad published Heart of Darkness, Roger Casement, then a British diplomat, revealed in a report that half of the population of Belgian-ruled Congo – nearly 10 million people – had perished under a brutal regime where beheadings, rape and genital mutilation of African labourers had become the norm. Such overt violence and terror is only a small part of the story of European domination of Asia and Africa, which includes the slow-motion slaughter of tens of million in famines caused by unfettered experiments in free trade – and plain callousness (Indians, after all, would go on breeding "like rabbits", Winston Churchill argued when asked to send relief during the Bengal famine of 1943-44).

The unctuous belief that British imperialists, compared to their Belgian and French counterparts, were exponents of fair play has been dented most recently by revelations about mass murder and torture during the British suppression of the Mau Mau uprising in Kenya in the 1950s. Nevertheless, in one of the weirdest episodes of recent history, a Kipling-esque rhetoric about bringing free trade and humane governance to "lesser breeds outside the law" has resonated again in the Anglo-American public sphere. Even before 9/11, Tony Blair was ready to tend, with military means if necessary, to, as he put it, "the starving, the wretched, the dispossessed, the ignorant" around the world. His apparently more intellectual rival Gordon Brown urged his compatriots to be "proud" of their imperial past. Sensing a sharper rightward shift after 9/11, many pith-helmet-and-jodhpurs fetishists boisterously outed themselves, exhorting politicians to recreate a new western imperium through old-style military conquest and occupation of native lands.

Embracing such fantasies of "full-spectrum dominance", American and European policymakers failed to ask themselves a simple question: whether, as Jonathan Schell put it, "the people of the world, having overthrown the territorial empires, are ready to bend the knee to an American overlord in the 21st"? After two unwinnable wars and horribly botched nation-building efforts, and many unconscionable human losses (between 600,000 and one million in Iraq alone), the "neo-imperialists" offering seductive fantasies of the west's potency look as reliable as the peddlers of fake Viagra. Yet, armour-plated against actuality by think tanks, academic sinecures and TV gigs, they continue to find eager customers. Of course, as the historian Richard Drayton points out, the writing of British imperial history, has long been a "patriotic enterprise". Wishing to "celebrate" empire, Michael Gove plans to entrust the task of rewriting the history syllabus to Niall Ferguson, one of the "neo-imperialist" cheerleaders of the assault on Iraq, who now craves "creative destruction" in Iran and whose "skilful revision of history" the Guardian's Jeevan Vasagar asserted last month, "will reverberate for years to come".

Clearly, it would help if no Asian or African voices interrupt this intellectual and moral onanism. Astonishing as it may seem, there is next to nothing in the new revisionist histories of empire, or even the insidious accounts of India and China catching up with the west, about how writers, thinkers and activists in one Asian country after another attested to the ravages of western imperialism in Asia: the immiseration of peasants and artisans, the collapse of living standards and the devastation of local cultures. We learn even less about how these early Asian leaders diagnosed from their special perspective the political and economic ideals of Europe and America, and accordingly defined their own tasks of self-strengthening.

Asian intellectuals couldn't help but notice that Europe's much-vaunted liberal traditions didn't travel well to its colonies. Mohammed Abduh, the founder of Islamic modernism, summed up a widespread sentiment when, after successive disillusionments, he confessed in 1895 that: "We Egyptians believed once in English liberalism and English sympathy; but we believe no longer, for facts are stronger than words. Your liberalness we see plainly is only for yourselves, and your sympathy with us is that of the wolf for the lamb which he deigns to eat."

In 1900, British atrocities during the Boer war and the brutal western suppression of the Boxer rising in China had provoked the pacifist poet Rabindranath Tagore to compare, in one unusually violent image, such bards of imperialism as Kipling to mangy dogs. "Awakening fear, the poet-mobs howl round / A chant of quarrelling curs on the burning-ground." Writing in 1907, the Indian nationalist Aurobindo Ghose was even harsher on lachrymose claims about the white man's burden. As Ghose saw it, previous conquerors, including the English in Ireland, had been serenely convinced that might is always right. But in the 19th century, the age of democratic nationalism, imperialism had to pretend "to be a trustee of liberty … These Pharisaic pretensions were especially necessary to British imperialism because in England the puritanic middle class had risen to power and imparted to the English temperament a sanctimonious self-righteousness which refused to indulge in injustice and selfish spoliation except under a cloak of virtue, benevolence and unselfish altruism."

There is something to Ghose's tirade. Free-traders and freebooters may have found merely convenient the idea that Asia was full of unenlightened people, who had to be saved from themselves. But many European and American intellectuals brought to it a solemn sincerity. Even John Stuart Mill, the patron saint of modern liberalism, claimed that "despotism is a legitimate mode of government in dealing with barbarians, if the end be their improvement." By 1900, such views had hardened into propaganda, and a mania for imperial expansion, drummed up by the press and politicians, had become part of the political life of European societies.

Scrambling to catch up with Europe, even the United States embraced the classic imperialism of conquest and occupation, expelling Spain from its Caribbean backyard and flexing its muscles in east Asia. In 1903, Liang Qichao, China's foremost modern intellectual and a major early influence on Mao Zedong, was visiting America when Washington manipulated its way into control of Panama and its crucial canal. It reminded Liang of how the British had compromised Egypt's independence over the Suez canal. Liang feared that original meaning of the Monroe doctrine – "the Americas belong to the people of the Americas" – was being transformed into "the Americas belong to the people of the United States". "And who knows," Liang added in a book he wrote about his travels, "if this will not continue to change, day after day from now on, into 'the world belongs to the United States'".

"In the world," Liang concluded bleakly, "there is only power – there is no other force … Hence, if we wish to attain liberty, there is no other road: we can only seek first to be strong." A whole generation of Chinese leaders and intellectuals grew up sharing Liang's social Darwinist belief "in the present-day international struggles in which the whole citizenry participate (and compete) for their very lives and properties, people are united as if they have one mind". No less a "westerniser" than Deng Xiaoping would uphold the primary imperative of national self-strengthening even as he broke with Maoism in the late 1970s and supervised China's transition to a market economy: "Our country must develop," Deng declared, using words emblazoned on billboards across China and still guiding the Communist party's politburo. "If we do not develop then we will be bullied. Development is the only hard truth."

Liang described the endless struggle between peoples enjoined by global capitalism as extremely dangerous. The first world war, which almost all European nations entered with great jingoistic fervour, following a period of hectic expansion, confirmed these anxieties. The poet and philosopher Muhammad Iqbal, who had spent three rewarding years as a student in Europe in the first decade of the 20th century, now wrote satirically of his old inspiration: "The West develops wonderful new skills / In this as in so many other fields / Its submarines are crocodiles / Its bombers rain destruction from the skies / Its gasses so obscure the sky / They blind the sun's world-seeing eye. / Dispatch this old fool to the West / To learn the art of killing fast – and best."

"European imperialism, which does not disdain to raise the absurd cry of the Yellow Peril," the Japanese art historian Kakuzo Okakura had written in 1906, "fails to realise that Asia may also awaken to the cruel sense of the White Disaster." In the wake of the first world war and the Paris peace conference, which inflicted cruel disappointments on India, China, Turkey, Egypt and Iran, many thinkers and activists in the east began to reconsider their earlier dalliance with western political ideals. Modernisation still seemed absolutely imperative, but it did not seem the same as westernisation, or to demand a comprehensive rejection of tradition or an equally complete imitation of the west. Freshly minted movements such as revolutionary communism and Islamic fundamentalism, which promised to immunise Asian countries against western imperialism, began to look attractive.

Europe's capacity and willingness for overseas expansion would be further diminished by an empire manqué – Germany – gone mad in its midst. Hitler turned out to be lethally envious of the British venture in India – what he called "the capitalist exploitation of the 350 million Indian slaves" – and hoped that Germany would impose a similarly kleptocratic despotism on the peoples and territories it conquered in Europe, while avoiding what he saw as Britain's lax racial segregation in India. "Nazism," Jawaharlal Nehru, India's first prime minister, shrewdly diagnosed in 1940, anticipating Hannah Arendt and other analysts of 20th-century European politics, was the "twin brother" of western imperialism, the latter functioning "abroad in colonies and dependencies, while fascism and Nazism functioned in the same way" within Europe.

For many people in Asia, the two world wars were essentially conflicts between Europe's rival empires rather than great moral struggles, as they were presented to European publics, between democracy and fascism – indeed, the long experience of imperialism made Asians experience the 20th century radically differently from their European overlords. Chafing at their degraded status in the white man's world, they were uniformly thrilled – Mohandas Gandhi, then an unknown lawyer in South Africa, as well as a young Ottoman soldier called Mustafa Kemal (later, Atatürk) – when in 1905 Japan defeated Russia. For the first time since the middle ages, a non-European country had vanquished a European power in a major war. And Japan's victory sparked a hundred fantasies – of national freedom, racial dignity, or simple vengefulness – in the minds of those who had sullenly endured European authority over their lands.

Gandhi correctly predicted that "so far and wide have the roots of Japanese victory spread that we cannot now visualise all the fruit it will put forth." Thirty-six years later, Japan struck the decisive blow to European power in Asia. In about 90 days beginning on 8 December 1941, Japan overran the possessions of Britain, the US and the Netherlands in east and south-east Asia, taking the Philippines, Singapore, Malaya, Hong Kong, the Dutch East Indies, much of Siam and French Indochina, and Burma with bewildering swiftness to stand poised at the borders of India by early 1942.

Shortly before Singapore fell to the Japanese in early 1942, the Dutch prime minister-in-exile, Pieter Gerbrandy, confided his anxiety to Churchill and other Allied leaders that "Japanese injuries and insults to the White population … would irreparably damage white prestige unless severely punished within a short time". After a long, hard struggle, the Japanese were finally "punished", fire- and nuclear-bombed into submission. The Japanese themselves behaved extremely brutally in many of the Asian countries they occupied. And yet, in the eyes of many Asians, the Japanese completely destroyed the aura of European power that had kept the natives in a permanent state of fear and political apathy.

Lee Kuan Yew, Singapore's founding father, recalled the lessons learnt by his generation of Asians: "that no one – neither the Japanese nor the British – had the right to push and kick us around". Accustomed to deferential natives, European powers mostly underestimated the post-war nationalism that the Japanese had both unwittingly and deliberately unleashed. They also misjudged their own staying power among populations unremittingly hostile to them. This led to many disastrously futile counter-insurgency operations and full-scale wars, especially in Indochina, which still scar large parts of Asia. Nevertheless, the speed of decolonisation was extraordinary.

Burma, which barely had a nationalist movement before 1935, became free in 1948. The Dutch in Indonesia resisted, but Indonesian nationalists led by Sukarno finally threw them out in 1953. Postwar chaos plunged Malaya, Singapore and Vietnam into prolonged insurgencies and wars, but the European withdrawal was never in doubt. A calamitous partition of the Indian subcontinent, which condemned two new nation-states to endless conflict, marked Britain's half-panicked departure in 1947; the following year, a similar combination of skulduggery and dereliction of duty in Palestine radically shrank the prospects for peace and stability in the Middle East.

Still, formal decolonisation, often accompanied by revolutions, transformed much of Asia and Africa in the 1950s and early 60s. Such leaders as Nehru, Mao, Nasser and Sukarno initially enjoyed great popularity and prestige, ostensibly engaged in the gigantic task of postcolonial consolidation – in Nehru's words, "What Europe did in 100 or 150 years, we must do in 10 or 15 years."

In contrast, "Europe," as Jean-Paul Sartre claimed in his strident preface to Franz Fanon's Wretched of the Earth, seemed to be "springing leaks everywhere". "In the past we made history," Sartre asserted, "and now it is being made of us." Watching Churchill's funeral in 1965, VS Pritchett felt an "undertone of grim self-pity" and premonitions of a "mean" future in which Britain would become to the larger world "one more irrelevant folk culture". But by the late 1960s, the massacre of communists in Indonesia, the intensified American assault on Vietnam, the overthrow of Nkrumah in Ghana and, finally, the election of Richard Nixon had made Hannah Arendt conclude that the "imperialist era", which seemed "half-forgotten", was "back, on an enormously enlarged scale".

The cold war, in which whoever was not with us was against us, had already distorted western views of Asia and Africa. The press of the "free world" was usually eager to assist the cold warriors define new enemies and allies. As Conor Cruise O'Brien described it, anti-communist liberals who dealt with the "sparse" news of brutal western puppets in Asia with "calm agnosticism" were prone to get very worked up over any signs of independent thinking among Asians. Indeed, as early as 1951, the New York Times had written off, in an editorial titled "The Lost Leader", the non-aligned Nehru as one of the "great disappointments to the post-war era".

Celebrating Independence Day in Karachi, Pakistan, 14 August 2011. Photograph: Daniel Berehulak/Getty Images

In his book The Myth of Independence (1969), the Pakistani leader Zulfikar Ali Bhutto warned his postcolonial compatriots that their "power to make decisions radically affecting the lives of our peoples" was being "curtailed by the cannons of neo-colonialism". Overthrown and murdered by a pro-American military despot, Bhutto was himself to exemplify what Ryszard Kapuscinski described as the tragic "drama" of many well-intentioned Asian and African leaders. Kapuscinski focused on the "terrible material resistance that each [leader] encounters on taking his first, second and third steps up the summit of power. Each one wants to do something good and begins to do it and then sees, after a month, after a year, after three years, that it just isn't happening, that it is slipping away, that it is bogged down in the sand. Everything is in the way: the centuries of backwardness, the primitive economy, the illiteracy, the religious fanaticism, the tribal blindness, the chronic hunger, the colonial past with its practice of debasing and dulling the conquered, the blackmail by the imperialists, the greed of the corrupt, the unemployment, the red ink. Progress comes with great difficulty along such a road. The politician begins to push too hard. He looks for a way out through dictatorship. The dictatorship then fathers an opposition. The opposition organises a coup. And the cycle begins anew."

The incompetence, corruption and brutality of many postcolonial leaders had become apparent by the end of the 1960s. Exhorting China to catch up with Britain's industrial output in less than a decade, Mao Zedong exposed tens of millions to a catastrophic famine, and then forced its exhausted survivors into a "cultural revolution". The extensive disorder of the postcolonial world, in which coups and civil wars became commonplace, made the age of European empires, when the unpoliticised natives knew their place, look peaceful in comparison.

Recoiling from absurd infatuations with third-worldism, even Maoism, on the left, many writers and intellectuals in Anglo-America began moving to the greener grass on the political right. A bien-pensant reaction to the 1960s also gathered strength (it was to culminate in our time in Sarkozy's and Blair's assaults on the decade's evidently dangerous "radical" consensus). In one sign of the reactionary climate of the 70s and 80s, Conor Cruise O'Brien, originally known for his exposé of western neo-colonialism in Africa, turned into a near-hysterical defender of apartheid in South Africa and the Israeli occupation of the West Bank and Gaza. It was also during these decades that VS Naipaul's withering accounts of "half-made" postcolonial societies came to be hugely influential.

Tracing Conrad's journey through the Congo, Naipaul claimed to see little difference between the imperialist and post-colonial eras. As he described it, the nihilism of Kurtz had been supplanted by "African nihilism, the rage of primitive men coming to themselves and finding that they have been fooled and affronted". Naipaul ignored cold-war machinations in the Congo just as he would later scant the brutal rule of Iran's shah in exchange for broad musings on the innate defects of Islam. Though quickly credited with ethnographic as well as literary authority, Naipaul offered mostly culturalist and pseudo-psychological generalisations – "Islam", for instance, was to blame for the incorrigible backwardness of Muslim countries, India was a "Wounded Civilisation" and of course "African nihilism" had done Africa in. These reductive accounts actually helped entrench, among even liberals, an ahistorical outlook on the non-west while confirming the western supremacist disdain for it. Speaking in 1990 to a rightwing think tank in New York, Naipaul evoked a widespread post-cold-war triumphalism by hailing the "universal civilisation" created by the west, which he claimed would blow away all rival ideologies and values.

Such was the aggressively self-congratulatory mood between the end of the cold war and 9/11: western-style democracy and capitalism stood poised not only to abolish the particularities of religion and culture but also to terminate history itself. Not surprisingly, al-Qaeda's attacks provoked yet more minatory readings of Islam as the irreconcilable foe of benign western liberalism rather than the long-delayed reckoning with the history of the west in the non-west and the divergent political and economic journeys of postcolonial countries.

As the Arab spring and its troubled aftermath shows, the long-delayed release from illusion and falsehoods in that part of the world will proceed from within; and it will be a long and arduous process. However, a similar effort to cleanse the west of imperial-age dogmas and attitudes has barely begun, as the recrudescence of a bellicose neo-imperialism in our time shows.

Could it be that Europe's abandonment of formal empire failed to provoke a cathartic revision of grandiose old notions of national and racial superiority? Certainly, projecting military force deep into Asia and Africa, Blair and Sarkozy seemed overly eager to borrow macho postures from the 19th century. Public nostalgia for the imperial era in Britain also continues to be tickled by patriotic historians, and "may appear", Drayton warns, "to be an innocent kind of solitary vice".

But the last decade of neo-imperialist "creative destruction" ruined, almost invisibly to its perpetrators and cheerleaders, millions of lives in remote lands. It is now obvious, as Drayton writes, that the intellectual "narcissism which orders the past to please the present" can also find "violent external expression in war and in an indifference towards the destruction, suffering and death of others".

Moreover, a narcissistic history – one obsessed with western ideals, achievements, failures and challenges – can only retard a useful understanding of the world today. For most people in Europe and America, the history of the present is still largely defined by victories in the second world war and the long standoff with Soviet communism, even though the central event of the modern era, for a majority of the world's population, is the intellectual and political awakening of Asia and its emergence, still incomplete, from the ruins of both Asian and European empires. The much-heralded shift of power from the west to the east may or may not happen. But only neo-imperialist dead-enders will deny that we have edged closer to the cosmopolitan future the first generation of modern Asian thinkers, writers and leaders dreamed of – in which people from different parts of the world meet as equals rather than as masters and slaves, and no one needs to shoot elephants to confirm their supremacy.

• Pankaj Mishra will be talking about his book From the Ruins of Empire: The Revolt Against the West and the Remaking of Asia at the LSE on 30 July at 6.30pm. More information at http://www2.lse.ac.uk/publicEvents.