Early this week a former senior official from Scotland Yard accused the Labour government of slavishly following the American neocon view of Islam and terrorism. According to him, the Anglo-American assault on Iraq and Afghanistan and British complicity in torture of suspects at Guantánamo greatly increased the risks of terrorist attacks on the UK, notwithstanding New Labour's strenuous attempts to deny such a causal connection. Reading his impassioned statement of the obvious, you could be forgiven for wondering why he, like many of the great and good at the Chilcot inquiry, hadn't spoken out previously – when it may have mattered.

The "heady poison" of war creates, as the US critic Randolph Bourne once put it, "its own antitoxin of ruin and disillusionment." Wars as misconceived as those in Iraq and Afghanistan – two of the longest in British and US history – are likely to generate many testimonies to folly and ineptitude. Lately in the US, there has been a flurry of confessions, accusations and counter-accusation from both officials who prosecuted the wars, and the intellectuals who cheerled them from the sidelines. The chastened mood, deepened by economic decline, is summed up by the title of a recent bestseller by Peter Beinart, one of the strident liberal "hawks": The Icarus Syndrome: A History of American Hubris.

What's remarkable about these long, tormented mea culpas is that they reveal little that the average newspaper reader does not already know. It was no secret that the invasion of Iraq had been conceived as a punitive expedition in the old-fashioned imperial style, in which economic and geopolitical aims were subordinate to the demonstration of firepower, to campaigns of shock and awe. As the US columnist Thomas Friedman, an ever-reliable barometer of official opinion, claimed: "What [Iraqis] needed to see was American boys and girls going house to house, from Basra to Baghdad, and basically saying, '… Suck. On. This.'" Nevertheless, men like Beinart concluded that the Bush administration was going to war for precisely the reason he and other intellectuals had insisted it ought to: to vanquish "Islamofascism" (never mind that Saddam Hussein was a secular despot) and thereby making the Middle East safe for liberal democracy.

As for Afghanistan, its modern history furnished plenty of cautionary tales against foreign invasions. Enraged by the 9/11 attacks, and seeking a suitably spectacular revenge, the US could be excused for not paying sufficient heed to them. But Britain, which has often fought unwinnable wars in far-off places for the sake of imperial pride, could have predicted that it would face a resilient Afghan adversary. It could draw upon the memory of three Afghan wars, which mostly proved disastrous even though it then did not have to worry about a significant Muslim population at home or such regional impediments as Pakistan and Iran.

Plenty of other historical examples – Napoleon in Spain, the French in Algeria, and the Americans in Vietnam – illustrate that a small but determined band of guerrilla fighters can annul the technical and often numerical superiority of foreign invaders. So what has persuaded Britain to remain embroiled in a hopeless imperial adventure in the early 21st century? After all, most ordinary citizens long turned against it, and their conviction of its futility hardens with every new casualty in Afghanistan. New Labourites weren't the only politicians to be neoconned or to be vulgarly infatuated with US power and wealth – the present education minister was among the busiest retailers of neocon fables. But the incompetence and failure of politicians and officials is not as shocking as the moral truancy of many intellectuals – the professionals paid to preserve and transmit historical and philosophical wisdom, and intelligently to interpret the contemporary world.

Perhaps, "it is a mistake," as Randolph Bourne wrote in his essay War and the Intellectuals, "to suppose that intellectuality makes for suspended judgments. The intellect craves certitude." Certainly, as recent history proves, dogma is hardly the exclusive malady of traditional religions – those that are fashionably blamed for all the evils in the world today. It is central, too, to the secular cults of our own time – progress, technology, military power – that many intellectuals keenly subscribe to.

Writing in the New York Times magazine a few weeks after the invasion of Iraq, the historian Niall Ferguson urged the US to impose capitalism and democracy through military force. Declaring himself as a "fully paid up member of the neoimperialist gang", Ferguson claimed to be fascinated by "the sort of enlightened foreign administration once provided by self-confident Englishmen in jodhpurs and pith helmets."

How risible these fantasies seem today, as America itself, enfeebled by jungle capitalism and feckless wars, looks as though it could benefit from a period of enlightened administration. And yet this me-too Lawrence-of-Arabia-ism was hailed not so long ago as grittily realistic in prestigious periodicals, and given a sympathetic hearing by political leaders, who, too, assumed that Asians and Africans, having recently sent their former European overlords home, were ready to kowtow to a new American master.

Of course the nostalgia for jodhpurs and pith helmets was so well received largely because it expressed itself through the resonantly noble rhetoric of humanitarianism. "I never knew a man," Graham Greene wrote in The Quiet American (1955), "who had better motives for all the trouble he caused." Indeed, Greene's portrait of a self-righteous American blundering through Vietnam holds up well as a general description of many do-gooding eggheads in our own time and the recent past.

Secular intellectuals with a passionately held faith in rational manipulation or simply itching to be in on the action led to the violent remaking of entire societies and cultures in the previous century: the colonisation of Asia and Africa as well as mass social engineering in Russia and China. They had little time for traditional religion, and scorned its old authority, which used to keep man's more Promethean lusts in check. Indeed, they were driven by a new religion: a belief in man's ability to radically reshape his social and natural environment.

Brute force was usually their means; according to the instrumentalist calculation, eggs have to be broken in order to make omelettes. The immense and uniquely ideological violence of the 20th century – what Camus described as the "slaves camps under the flag of freedom, massacres justified by philanthropy" – partly derived from a mode of reasoning that made the vast and unpredictable realm of human affairs appear as amenable to radical mutation as a box of eggs does at breakfast time.

Many such "rational" ideologies now lie in the dustbin of history; but the illusions of omnipotence continue to flourish, and their first eager victims still tend to be intellectuals trying to secure a footnote, if not a whole chapter, for themselves in history. Bookish do-gooders drunk on abstractions may not actually appear to be among the most menacing figures of our time – at least not when compared to bushy-bearded fanatics, moustachioed despots and smooth-shaven elected leaders brimming with conviction. But they, too, have been doing their bit, if quietly, to make the new century as bloody as the previous one.