We are winning in Iraq, John McCain declared in the presidential debate last week, "and we will come home with victory and with honour." This may sound like some perfunctory keep-the-pecker-up stuff from a former military man. But the Republican candidate, who believes that the "surge" has succeeded in Iraq, also possesses the fanatical conviction that heavier bombing and more ground troops could have saved the United States from disgrace in Vietnam.

On the same occasion, Barack Obama, who seems more aware of the costs of American honour to the American economy, claimed he would divert troops from Iraq to Afghanistan and, if necessary, order them to assault "safe havens" for terrorists in Pakistan's wild west. Both candidates sought the imprimatur of Henry Kissinger, the co-alchemist, with Richard Nixon, of the "peace with honour" formula in Vietnam, which turned out to include the destruction of neighbouring Cambodia.

An ominously similar escalation of the "war on terror" has ensured that the next American president will receive a septic chalice from George Bush in January 2009. In July, Bush sanctioned raids into Pakistan, pre-empting Obama's tough-sounding strategy of widening the war in Afghanistan, where resurgent Taliban this year account for Nato's highest death toll since 2001. Pakistan's army chief vowed to defend his country "at all costs", and his soldiers now clash with US troops almost daily. Obscured by the American economy's slow-motion train wreck, the war on terror has already stumbled into its most treacherous phase with the invasion of fiercely nationalistic and nuclear-armed Pakistan.

Most of the recent disasters of geopolitical machismo could have been foretold. In late 2003, when the occupation of Iraq was beginning to go badly wrong, the American journalist Dexter Filkins came across a village called Abu Hishma in the Sunni triangle. Rubble-strewn and "encased in razor wire", Abu Hishma resembled, Filkins writes in The Forever War: Dispatches from the War on Terror (Bodley Head), "a town in the West Bank". Its terrified residents told him about the local American commander Nathan Sassaman, who bulldozed homes and called in air strikes, and who was fond of proclaiming that "there is no God - I am god here".

Sassaman sounds like something out of Conrad, the white man in the tropics driven to lunacy by absolute power and extreme isolation. But, according to Filkins, he is a bright man, even the "embodiment of the best that America could offer" in his desire to bring democracy to Iraqis. A serious reader of history and anthropology, Sassaman, along with fellow officers, is very impressed by a book entitled The Arab Mind, by Raphael Patai, a Hungarian-Israeli-American academic. Apparently, it makes clear that the "only thing" the denizens of the Middle East "understand is force - force, pride and saving face", and Sassaman believes that, "with a heavy dose of fear and violence, and a lot of money for projects ... we can convince these people that we are here to help them".

Filkins doesn't mention that The Arab Mind, originally published in 1973, was the bible of neocon commentators in Washington and New York cheerleading the Bush administration's audacious venture: what Condoleezza Rice in the new book by Bob Woodward, The War Within: A Secret White House History 2006-2008 (Simon & Schuster), describes as shifting the "epicentre of American power" from Europe, where it had rested since the second world war, to the Middle East. Widely read in the US military, The Arab Mind later inspired the modus operandi of the jailers of Abu Ghraib.

More surprisingly, respectable intellectuals, journalists and academics echoed its generalisations. Among these people was the historian Bernard Lewis, who assured Dick Cheney, one of his most devoted readers, that "in that part of the world, nothing matters more than resolute will and force". The New York Times columnist Thomas Friedman (who is on Sassaman's reading list) exhorted the US to act "just a little bit crazy", since "the more frightened our enemies are today, the fewer we will have to fight tomorrow". Accordingly, Richard Armitage, assistant secretary of state and a relative moderate among the Bush administration's hawks, told Pakistani diplomats that the US would bomb their country "back to the stone age" if it did not withdraw its support for the Taliban.

The idea that the natives would recognise superior firepower when they saw it seemed to be validated by Pakistani acquiescence, followed by the Taliban's swift capitulation. Iraq was logically the next setting for shock-and-awe tactics - Donald Rumsfeld was complaining even before the aerial bombing of the Taliban had finished that Afghanistan had run out of targets. The Bush administration claimed that Saddam Hussein had to be disarmed to make the Middle East safe for democracy. But invading Iraq was also an image-making exercise - what Hannah Arendt, commenting on the absence of clear military goals in America's previous war of choice in Vietnam, described as the attempt by "a superpower to create for itself an image which would convince the world that it was indeed 'the mightiest power on earth'".

Busy unleashing his awesome firepower on Iraq, Rumsfeld had no idea what to do after his streamlined army reached Baghdad, apart from letting stuff happen. Wiser in Battle, the memoir of the US lieutenant general Ricardo Sanchez (HarperCollins), reveals that, as the Iraqi resistance unexpectedly intensified, the defeat in Vietnam began to prey on Bush's mind, unravelling his syntax as he harangued his commanders in Iraq:

Kick ass! ... We must be tougher than hell! This Vietnam stuff, this is not even close. It is a mind-set. We can't send that message. It's an excuse to prepare us for withdrawal ... There is a series of moments and this is one of them. Our will is being tested, but we are resolute. We have a better way. Stay strong! Stay the course! Kill them! Be confident! Prevail! We are going to wipe them out! We are not blinking!

Thomas Ricks, the Washington Post's Pentagon correspondent, describes in his book Fiasco (Penguin) how, after a mob ambushed and killed four American military contractors in Falluja, the commanders were ordered to "go in and clobber". Citing strategic and logistical reasons, the military chiefs pleaded for restraint, but they were overruled by the White House: the destruction of Falluja was as essential to the image-making exercise as the carpet-bombing of Vietnam and Cambodia.

The geopolitical consequences as well as the "collateral" damage of the exhibition of US might are succinctly outlined by the titles of recent books - The Forever War, Fiasco and Ahmed Rashid's Descent into Chaos (Allen Lane). Rashid is clearly the most despairing among the journalists accompanying the march of folly, even though, as a Pakistani long accustomed to the pretensions and limits of US power in south Asia, he didn't start off with many illusions. His previous book described how a combination of selfish motives and reckless actions by the US facilitated the rise of Islamic fundamentalism in Pakistan and Afghanistan. "Outsiders like me," he writes in Descent into Chaos, "found it remarkable that a US president could live in such an unreal world, where the entire military and intelligence establishments were so gullible, the media so complacent, Congress so unquestioning - all of them involved in feeding half-truths to the American public."

The habitual deceivers are often, in the end, the most deceived. According to Rashid, Pervez Musharraf's regime in Pakistan may have pulled off one of the biggest swindles in recent history by persuading the Bush administration to part with $10bn in exchange for mostly empty promises of support for its "war on terror". Most Pakistanis feel a mix of contempt and distrust for the US, which abandoned their country after enlisting it in a proxy war against the Soviet Union in Afghanistan. Confronted with a choice between regressing to the stone age and meeting crazy Uncle Sam's demands, Musharraf's regime adopted a policy of dissembling that the then foreign minister outlined as "First say yes, and later say but". Since 9/11, the Inter-Services Intelligence (ISI), Pakistan's rogue spy agency, which has long considered Afghanistan as its backyard, has continued to provide sanctuary and military support for the Taliban while occasionally arresting some al-Qaida militants to appease Washington. Mullah Omar and the original Afghan Taliban Shura, Rashid claims, are serenely resident in Pakistan's borderlands, along with "a plethora of Asian and Arab terrorist groups who are now expanding their reach into Europe and the United States".

"I'm not," Bush said soon after 9/11, "going to fire a $2m missile at a $10 empty tent and hit a camel in the butt." Hitting camels in the butt may have been more useful than disbursing $70m in bribes to warlords such as Abdul Rashid Dostum, whom Rashid revealed in his previous book to be fond of driving tanks over his opponents. The US coaxed many of Afghanistan's old villains out of retirement to defeat the Taliban with minimum use of US troops, and then lost interest in the country.

Rashid believes that the US could have done more to help "nation-building" in Afghanistan or at least prop up Hamid Karzai, who last week was reduced to plaintively asking Mullah Omar to return to Afghanistan for the sake of "peace". But as Tariq Ali bluntly clarifies in his new book The Duel: Pakistan on the Flight Path of American Power (Simon & Schuster), the post-9/11 project of "nation-building" in Afghanistan, which prioritised western interests over all others, was always doomed. It was "a top-down process", trying to create "an army constituted not to defend the nation but to impose order on its own people, on behalf of outside powers; a civil administration that will have no control over planning, health, education etc, all of which will be run by NGOs, whose employees will be far better paid than the locals, and answerable not to the population but to their overseas sponsors; and a government whose foreign policy is identical to Washington's."

American bombing raids, which have killed hundreds of civilians in Afghanistan, further unite fractious Afghans against foreign usurpers. Tariq Ali correctly prescribes scepticism against strategists and journalists who blame Pakistan for increasing attacks on western forces in Afghanistan while disregarding the fact that "many Afghans who detest the Taliban are so angered by the failures of Nato and the behaviour of its troops that they will support any opposition."

In Pakistan, too, public anger against the US is fuelled largely by the "knowledge that Washington has backed every military dictator who has squatted on top of the country". Contemptuously dismissing the alarmist cliché that jihadis are very close to getting their grubby fingers on the country's nuclear button, Ali points to the deep and persistent unpopularity of religious parties in Pakistan. The jihadis would only get that far, he asserts, if "the army wanted them to", which is virtually impossible unless, as may be beginning to happen now, American assaults on the country's hard-won sovereignty causes deep ideological ruptures within the country's strongest institution.

Filkins doesn't set out any future trajectory for the venture in Iraq. He reported from the country for the New York Times, but the first-person narrator of The Forever War is less a journalist than an existential hero, eloquent with the pathos of Sisyphean striving, impotence and failure. Composed in short, often lyrical, sections, Filkins's book often seems aimed at literary posterity, where it would join such modern classics of war literature as Ernst Junger's Storm of Steel, André Malraux's La Condition Humaine and Michael Herr's Dispatches

Unlike the war in Vietnam, which exercised some of the keenest literary sensibilities in America (Norman Mailer, Mary McCarthy, Susan Sontag), the entanglement in Iraq and Afghanistan has produced, so far at least, a meagre crop of quality journalism. The Forever War, which generally eschews historical overviews and extended analysis, succeeds more than most recent books in making cinematically vivid and imaginatively coherent the many places of horror and bewilderment that Americans have stumbled into during the "war on terror". This is what it is like - its brief confessions of doubt, confusion, panic and weariness seem to say - for an American young man witnessing the terrible violence of places such as Iraq and Afghanistan.

Filkins has the exasperation of the well-travelled and atrocity-hardened journalist with his sheltered compatriots back home. The attack on the twin towers in New York makes him think that "I was back in the third world ... My countrymen are going to think this is the end of the world, the worst thing that ever happened. In the third world this sort of thing happens every day." After the cyclone in Orissa, Filkins writes, "the dead were piled so high and for so long that the dogs couldn't eat any more". But as he strives to share with us the cruelty and tragedy of the world outside the west, he inadvertently reveals his historical innocence as well as hinting at a wider incomprehension of the postcolonial world and the decolonised mind.

"Do Americans imagine," Jonathan Schell once wrote, "that the people of the world, having overthrown the territorial empires, are ready to bend the knee to an American overlord in the 21st?" Even at his most tormented, Filkins doesn't really ask this crucial question, though he cannot help but be aware of the Iraqi people's eagerness to see the back of their "liberators" - to "tell the Americans what they want to hear and they will go away, and we can carry on the way we want". In a brilliantly paced account of Falluja, Filkins describes Iraqi children recoiling from American candy as if it were "radioactive" and remarks that the city "was like that from the start, even before the big battle in November 2004. Anything the Americans tried there turned to dust." This may convey well to an American audience the frustration of their do-gooding representatives in Iraq who alternate bombs with candy. But Filkins would have advanced a greater appreciation of national or tribal feeling in Iraq if he had explained that residents of Falluja were equally intransigent in 1920, when the British imperial army had to destroy the city in order to save it.

As perplexed as the British once were about Iraq ("so complex, its ways so labyrinthine"), Filkins writes with obvious fascination about Ahmed Chalabi, the Iraqi expatriate who managed to deceive some of the shrewdest politicians and journalists in America. In many ways, Chalabi, a chronic conspirator with mysterious allies in Iran, vindicates John Quincy Adams's warning to his young nation in 1821 against European-style imperialist adventures: by going "abroad", Adams wrote, "in search of monsters to destroy", America would "involve herself beyond the power of extraction in all the wars of interest and intrigue, of individual avarice, envy and ambition". But Chalabi, for Filkins, embodies Iraq's vexations rather than American blundering. "When I looked," he writes, "into Chalabi's eyes and saw the mirrors and doors closing, I knew that I was seeing not just the essence of the man but of the country to which he'd returned. L'etat c'est lui. Chalabi was Iraq."

Whatever may be said about this amiable fraud, he was certainly not Iraq - the country he had barely set foot in for more than 40 years before the US invasion. Though easily the most original and intense American book yet on the war in Iraq, The Forever War is far from matching the stupendous achievement of Neil Sheehan's A Bright Shining Lie, which expresses more resonantly the American reporter's angst while ruthlessly investigating specific national flaws - racism, cold war paranoia, belief in technology - that entrapped the US in Vietnam. By abandoning the tasks of analysis and introspection, Filkins's book makes us suspect that the "forever war" might make a new generation of can-do Americans weary, but not wiser.

"I'd gotten caught up," he writes at one point, "in the trappings and the pronouncements of officialdom, Iraqi and American." It is an admirably honest confession, but not one that the best US journalists covering their country's last big war would have made. Arriving in Vietnam in the early 1960s to report what was then a covert US operation, Homer Bigart, David Halberstam, Charley Mohr and Sheehan spent only a few weeks in the fog of official bluster. Their boldness was especially remarkable because the US media in the 1950s had largely shared a consensus about the dangers of communism with the White House and State Department, which regarded even newspapers such as the New York Times as extensions of foreign policy.

Ignoring the adversarial standard set by Halberstam and Sheehan, many US journalists and commentators in the post-cold-war era have been too eager to uphold their government's claims. It is not clear if Filkins was one of the liberal interventionists with a naive faith in the Bush administration's promise to promote democracy through war. Certainly his frequent laments about the inept handling of the invasion and occupation of Iraq do not lead him to conclude that an intrinsically bad idea could have been handled much differently. He can also permit himself the belief that "perhaps in the hideous present some larger good was being born".

This has the unfortunate echo of Condoleezza Rice's assertion, as Israel devastated Lebanon in 2006, that we were witnessing the "birth pangs of a new Middle East": that is, the Lebanese, bombed into a revolt against Hizbullah, would make their country safe for pro-Israeli and pro-American democracy. Totalitarian regimes and terrorist groups commonly use violence as a means to large-scale political engineering; it is more remarkable when democratic countries such as Israel and America do so, usually in flagrant disregard of the lessons of contemporary history. In A Choice of Enemies: America Confronts the Middle East (Weidenfeld & Nicolson), Lawrence Freedman describes in detail the awful results - the massacre of hundreds of Palestinian refugees, the rise of Hizbullah - of Israel's previous attempt in 1982 to redraw the map of the Middle East by assaulting Lebanon.

It is one thing for a small country with a perennial existential crisis to believe that, as the Israeli general Moshe Dayan once put it, "it was in our power to set a high price on our blood, a price too high for the Arab community, the Arab army or the Arab governments to think it worth paying". But how did the US let its foreign policy become hostage to a strategy of pre-emptive war and brutal retaliation? "How," Freedman asks, "had the United States gotten itself in this position, entangled in the confusing and often violent geopolitics of the Middle East and beset by enemies on all sides."

This is a bigger story than anyone can tell in one book, and the 600 pages of A Choice of Enemies cover only US foreign policy decisions from 1979, with the Islamic revolution in Iran and the Soviet invasion of Afghanistan. Freedman is forced to skim some important details of the relationship between the US and Israel, whose continuing expansion into the occupied West Bank is probably the greatest source of the so-called Arab rage. The State Department in 1948 argued passionately against supporting a Jewish state in Palestine. The Eisenhower administration, which saw Israel as an irritant, undermining the US alliance with anti-Soviet regimes in the Middle East, ensured that the joint Anglo-French-Israeli attack on Egypt would fail. John F Kennedy sent feelers to Egypt's fiercely anti-Zionist president Gamal Abdel Nasser. Lyndon B Johnson was the first US president to manipulate foreign policy in order to bolster Jewish-American support for the Democratic party; but even he was not able to build his "special relationship" with Israel without encountering strong opposition from American diplomats.

"I could not believe what I was hearing," Jimmy Carter wrote in his diary after Menachem Begin confided in him his desire to reduce Palestinians on the West Bank to a minority. Even Ronald Reagan, who believed that God fixed the Middle East as the site of Armageddon, stuck to a cold war policy of close relations with reliably anti-Soviet and oil-rich Arab regimes. Friendly to Saudi Arabia, Bush Sr was actively hostile to Israeli expansionism. His secretary of state, James Baker, had only blunt wisdom ("Forswear annexation. Stop settlement activity. Reach out to Palestinians as neighbours who deserve political rights") to impart to the American Israel Public Affairs Committee (AIPAC), the powerful lobbying outfit for Israel, to which even Obama must now genuflect.

Israel played a very small role in the blunders US administrations made in the late 1970s: to support the Shah of Iran long after his rule became widely despised and unsustainable, and, more fatefully, to mobilise a global Islamic jihad against Soviet communism. Trying to turn Afghanistan, as Carter's national security adviser, Zbigniew Brzezinski, exulted, into the "Soviet Union's own Vietnam", the CIA chose Pakistan as a logistical base for its anti-communist jihad. It funnelled money and arms to the ISI, which in turn passed on some of them to its own Islamist protégés (some of whom are now fighting US and Nato troops in Afghanistan). Radicals from Saudi Arabia, Egypt, Algeria and other Arab countries flocked to Pakistan to contribute to the holy war against atheistic communism. Freedman's statistics are a grim reminder of how the blowback from this first global jihad wrecked Pakistan long before it crashed into the west on September 11 2001. Pakistan, which had 900 religious schools in 1971, had "about 8,000 official and as many as 25,000 unregistered madrassas" by 1988.

But if US officials noticed their indirect sponsorship of radical Islam, they did not care. As Freedman writes, "the Reagan administration associated terrorism with leftist, secular groups linked to nationalist movements, whether the Irish, Basques or Palestinians ... Arab militants coming to support the jihad were seen at most as the equivalent of the idealists of the 1930s who joined the International Brigades during the Spanish civil war."

Freedman recalls, too, some of the other political machinations that have come to haunt present generations: US support for Iraq in the latter's long war against Iran, which emboldened Saddam Hussein all through the 1980s, and the deliberate indifference to the victims of Iraq's chemical warfare. At least in these and other cynical moves, the US could claim the sanction of realpolitik. Great powers often have to make unpleasant choices to protect their interests; they have also been known occasionally to thrash (Reagan in Grenada, Putin in Georgia) a pesky neighbour or two. What is startlingly new is the Bush administration's experiment of intimidating entire peoples as well as governments in the Middle East into accepting America's worldwide hegemony.

Seven years on, hundreds of thousands are dead, and millions of refugees on the move, while the US seems only to have boosted its old enemies in Afghanistan, Iran and Lebanon, and created formidable new ones in Iraq and Pakistan. In The War Within, Woodward shows the US president slipping deeper into his own world. "We're killin' 'em! We're killin' 'em all!" But not even the Bush administration, which has proved ready to do unspeakable things to its perceived enemies, can kill them all. It can continue to stage elaborate shock-and-awe spectacles, but if, as is increasingly evident, the target audience refuses to be impressed by them, they are rendered utterly futile - even dangerously counterproductive. "Force," as James Baldwin pointed out in the early 1970s during the US bombing of Indochina, "does not work the way its advocates seem to think it does. It does not, for instance, reveal to the victim the strength of his adversary. On the contrary, it reveals the weakness, even the panic of his adversary and this revelation invests the victim with patience."

Apparently routed by heavy B-52 bombing in 2001, the Taliban are resurgent, straining the military resources of the US and Nato in Afghanistan to the limit. In Iraq the strategy based on overwhelming force has proved to be a catastrophic failure, and had to be replaced by General David Petraeus's new counter-insurgency doctrine that emphasises political over military tactics. Nevertheless, McCain pledges "victory" in Iraq, whatever that takes, including a 100-year-long military presence in the country; threatening Russia, he also seems ready to bomb Iran. Obama, though keen to withdraw troops from Iraq, upholds the complacently bipartisan consensus about Afghanistan. But more US troops in the Pashtun heartland may merely underscore the lesson learned at a terrible cost by the British army in 1839-42 and 1878-80, and the Soviet army in 1979-89. "A troop increase," Rory Stewart recently reiterated in Time magazine, "is likely to inflame Afghan nationalism because Afghans are more anti-foreign than we acknowledge, and the support for our presence in the insurgency areas is declining." The violation of Pakistan's sovereignty threatens to be the most calamitous of recent US misadventures.

A gracious acceptance of the limits of US firepower may not be forthcoming from the next administration, which will face the hard choice to get out or fight on. Indeed, failure may make it even more determined to maintain the pride of US arms and the image of the mightiest power on earth. The prospect of humiliation in Vietnam was what prompted Nixon's devastation of Cambodia, setting the stage for the genocidal Pol Pot. As Hannah Arendt wrote, "when all signs pointed to defeat", the goal was "no longer one of avoiding humiliating defeat but of finding ways and means to avoid admitting it and 'save face'."

Could smashing up Iran or invading Pakistan become the face-saving formula for the exponents of "shock and awe"? Certainly, they see US force impressing the Persian and the Pakistani mind as it apparently has the Arab mind. And such is the crazy logic of a wounded militarism that, notwithstanding its battered economy, the US may soon be embattled on many more fronts in what is already its most damaging war.