The United States today spends approximately as much as the rest of the world combined on its military establishment. So it is worth pondering why it is that, after nearly four years of effort, the loss of thousands of American lives, and an outlay of perhaps half-a-trillion dollars, the US has not succeeded in pacifying a small country of some 24 million people, much less in leading it to anything that looks remotely like a successful democracy.

One answer is that the nature of global politics in the first decade of the 21st century has changed in important ways. Today's world, at least in that band of instability that runs from north Africa and through the Middle East, sub-Saharan Africa and central Asia, is characterised by numerous weak and sometimes failed states, and by transnational actors who are able to move fluidly across international borders, abetted by the same technological capabilities that produced globalisation. States such as Afghanistan, Pakistan, Iraq, Lebanon, Somalia, Palestine and a host of others are not able to exercise sovereign control over their territory, ceding power and influence to terrorist groups such as al-Qaida, political parties-cum-militias such as Hizbullah in Lebanon, or various ethnic and sectarian factions elsewhere.

American military doctrine has emphasised the use of overwhelming force, applied suddenly and decisively, to defeat the enemy. But in a world where insurgents and militias deploy invisibly among civilian populations, overwhelming force is almost always counterproductive: it alienates precisely those people who have to make a break with the hardcore fighters and deny them the ability to operate freely. The kind of counterinsurgency campaign needed to defeat transnational militias and terrorists puts political goals ahead of military ones, and emphasises hearts and minds over shock and awe.

A second lesson that should have been drawn from the past five years is that preventive war cannot be the basis of a long-term US nonproliferation strategy. The Bush doctrine sought to use preventive war against Iraq as a means of raising the perceived cost to would-be proliferators of approaching the nuclear threshold. Unfortunately, the cost to the US itself was so high that it taught exactly the opposite lesson: the deterrent effect of American conventional power is low, and the likelihood of preventive war actually decreases if a country manages to cross that threshold.

A final lesson that should have been drawn from the Iraq war is that the current US government has demonstrated great incompetence in its day-to-day management of policy. One of the striking things about the performance of the Bush administration is how poorly it has followed through in accomplishing the ambitious objectives it set for itself. In Iraq, the administration has acted like a patient with attention-deficit disorder. The US succeeded in organising efficiently for key events such as the handover of sovereignty on June 30 2004, or the elections of January 30 2005. But it failed to train Iraqi forces, failed to appoint ambassadors, failed to perform due diligence on contractors and, above all, failed to hold accountable those officials most responsible for these and other multiple failures.

This lack of operational competence could in theory be fixed over time, but it has important short-term consequences for American grand strategy. Neoconservative theorists saw America exercising a benevolent hegemony over the world, using its enormous power wisely and decisively to fix problems such as terrorism, proliferation, rogue states, and human-rights abuses. But even if friends and allies were inclined to trust America's good intentions, it would be hard for them not to be dismayed at the actual execution of policy and the amount of broken china this particular bull left behind.

The failure to absorb Iraq's lessons has been evident in the neoconservative discussion of how to deal with Iran's growing regional power, and its nuclear programme. Iran today constitutes a huge challenge for the US, as well as for America's friends in the Middle East. Unlike al-Qaida, Iran is a state, deeply rooted historically (unlike Iraq) and flush with resources as a result of energy price rises. It is ruled by a radical Islamist regime that - particularly since Mahmoud Ahmadinejad's election in June 2005 - has turned in a disturbingly intolerant and aggressive direction.

The US unintentionally abetted Iran's regional rise by invading Iraq, eliminating the Ba'athist regime as a counterweight, and empowering Shia parties close to Tehran. It seems reasonably clear that Iran wants nuclear weapons, despite protestations that its nuclear programme is only for civilian purposes; nuclear energy makes little sense for a country sitting on some of the world's largest oil reserves, but it makes sense as the basis for a weapons programme. It is completely rational for the Iranians to conclude that they will be safer with a bomb than without one.

It is easy to outline the obstacles to a negotiated end to the Iranian programme, but much harder to come up with an alternative strategy. Use of force looks very unappealing. The US is hardly in a position to invade and occupy yet another country, especially one three times larger than Iraq. An attack would have to be conducted from the air, and it would not result in regime change, which is the only long-term means of stopping the WMD programme. It is hard to have much confidence that US intelligence on Iranian facilities is any better than it was in the case of Iraq. An air campaign is much more likely to build support for the regime than to topple it, and will stimulate terrorism and attacks on American facilities and friends around the globe. The US would be even more isolated in such a war than during the Iraqi campaign, with only Israel as a certain ally.

None of these considerations, nor the debacle in Iraq, has prevented certain neoconservatives from advocating military action against Iran. Some insist that Iran poses an even greater threat than Iraq, avoiding the fact that their zealous advocacy of the Iraq invasion is what has destroyed America's credibility and undercut its ability to take strong measures against Iran.

All of this could well be correct. Ahmadinejad may be the new Hitler; the current negotiations could be our Munich accords; Iran could be in the grip of undeterrable religious fanatics; and the west might be facing a "civilisational" danger. I believe that there are reasons for being less alarmist. Iran is, after all, a state, with equities to defend - it should be deterrable by other states possessing nuclear weapons; it is a regional and not a global power; it has in the past announced extreme ideological goals but has seldom acted on them when important national interests were at stake; and its decision-making process appears neither unified nor under the control of the most radical forces.

What I find remarkable about the neoconservative line of argument on Iran, however, is how little changed it is in its basic assumptions and tonalities from that taken on Iraq in 2002, despite the momentous events of the past five years and the manifest failure of policies that neoconservatives themselves advocated. What may change is the American public's willingness to listen to them.

· This is an edited extract from After the Neocons by Francis Fukuyama, published in paperback by Profile books at £7.99