On November 19 2002, four months before the invasion of Iraq, Tony Blair made a rare attempt to seek out expert views beyond the circle of his official advisers. Six distinguished academics were invited to Downing Street: three specialists on Iraq, and three on international security. George Joffe, an Arabist from Cambridge University, and Charles Tripp and Toby Dodge, who had both written books on Iraq's history, made opening statements of about five minutes each. They decided not to alienate the prime minister by discussing whether an invasion was sensible or necessary, but only what its consequences might be.

"We all pretty much said the same thing," Joffe recalls. "Iraq is a very complicated country, there are tremendous intercommunal resentments, and don't imagine you'll be welcomed." He remembers how Blair reacted. "He looked at me and said, 'But the man's uniquely evil, isn't he?' I was a bit nonplussed. It didn't seem to be very relevant." Recovering, Joffe went on to argue that Saddam was constrained by various factors, to which Blair merely repeated his first point: "He can make choices, can't he?" As Joffe puts it, "He meant he can choose to be good or evil, I suppose."

Joffe got the impression of "someone with a very shallow mind, who's not interested in issues other than the personalities of the top people, no interest in social forces, political trends, etc".

Dodge also struggled to convince Blair of the obstacles that would face anyone who occupied Iraq. "Much of the rhetoric from Washington appeared to depict Saddam's regime as something separate from Iraqi society," he remembers. "All you had to do was remove him and the 60 bad men around him. What we wanted to get across was that over 35 years the regime had embedded itself into Iraqi society, broken it down and totally transformed it. We would be going into a vacuum, where there were no allies to be found, except possibly for the Kurds."

The experts didn't seem to make much of an impression. Blair "wasn't focused", Tripp recalls. "I felt he wanted us to reinforce his gut instinct that Saddam was a monster. It was a weird mixture of total cynicism and moral fervour."

The brief meeting was unique. "I can't remember participating in any meaningful seminar on Iraq with the Foreign Office," Tripp says. "We were not asked to brief officials in the Middle East department."

What has since become clear is that Joffe, Dodge and Tripp were not the only experts to be left out in the cold. In April 2004, after a weekend in which rockets, helicopter attacks and shootings left dozens of Iraqis dead, 52 retired British diplomats, most of them career specialists on the Middle East, wrote an extraordinary open letter to Blair deploring Britain's lack of proper prewar analysis. They described Iraq as the region's most complex country and said it was naive for the Americans and British to think they could create a democratic society, however much some Iraqis might want one.

"All those with experience of the area predicted that the occupation of Iraq by the coalition forces would meet serious and stubborn resistance, as has proved to be the case. To describe the resistance as led by terrorists, fanatics and foreigners is neither convincing nor helpful," they declared.

The letter caused a political sensation. Retired diplomats do not often go on record in such direct opposition to their former employer, nor in such numbers. Here was the voice of the Foreign Office's senior Arabists, ranged against a prime minister who did not understand the region.

At the time, many analysts assumed the writers' views were shared by their colleagues still in government service. What the 52 were saying must surely be an on-the-record distillation of what the Foreign Office's officials were telling Jack Straw, their minister, and Downing Street in private.

Astonishingly, this was not the case. Interviews with top Foreign Office officials involved in the prewar discussions as well as Arabic-speaking British ambassadors in the region reveal a damaging vacuum in the department's advice. The predictions that the 52 claimed were made by "all those with experience of the area" may have been shared privately inside the Foreign Office's grand Italianate mansion in Whitehall, but they did not circulate as official thinking or reach ministers. While some senior officials in Britain's intelligence agencies expressed their doubts that Saddam was genuinely stockpiling weapons of mass destruction, no serious qualms were raised by the government's foreign policy experts about the equally important problem of whether occupying Iraq could work. Analysing the likely consequences of invading one of the major Arab states should have been a crucial element in judging whether it was in Britain's interest, let alone that of ordinary Iraqis, to go to war. Yet such analysis was simply absent. Ministers never asked for it; officials never offered it.

Neither of the Foreign Office's top two officials, Sir Michael Jay, the permanent under-secretary, and Sir Peter Ricketts, the political director, were Arabists. In Downing Street Blair's top foreign policy adviser, Sir David Manning, had been Britain's ambassador in Israel from 1995 to 1998 but never served in an Arab capital. No one in Whitehall rang alarm balls by recalling the difficulties of Britain's imperial involvement with Iraq and the long years of resistance to British occupation, particularly in the largely Shia south. British colonisers had invaded the country, defeated the Ottoman army, and assumed total control in 1918. They abolished the elected municipal councils, imposed a foreign Sunni monarchy, and dealt with resistance by means of massive military repression. Weak on Iraqi history, officials were also poor in forecasting future scenarios. No one pointed out that Saddam's removal would very probably give a boost to Shia Islamists and strengthen the Islamist parties that were allied to Iran. This would make nonsense of hopes for Iraq to become pro-western while remaining, as Saddam's Iraq was, a bastion against the mullahs in Tehran.

If the government ever answers calls for a full-scale inquiry into the policy discussions that led to the invasion of Iraq, there is a danger that it will focus on WMDs, or blunders such as the failure to control mass looting or the decision to dissolve the Iraqi army. But what about the serious lapses in political analysis? It is often argued that the occupation stumbled because of a lack of prewar planning, but the real problem was a failure to comprehend that western armies cannot successfully take over Arab countries and force them to run along western lines. The occupation was doomed from the start. No matter how efficient, sensitive, generous and intelligent the Coalition Provisional Authority (CPA) had been, it could not have succeeded. Occupations are inherently humiliating. People prefer to run their own affairs; they resent foreigners taking over their country. A foreign army that topples a regime needs to leave within weeks or at most months. Otherwise, suspicion will grow quickly that the foreigners' real aims are imperial - to run the country directly or through the locals they put in charge, and to exploit its resources. Nowhere is this truer than in the Middle East, where feelings of dignity, honour, sovereignty and humiliation are the currency of daily life.

With blithe self-confidence, and without even asking his officials for expertise, however, Blair assumed it would be easy for the US and UK to run the country after Saddam was toppled. His style was not to encourage his policy preferences to be questioned, or call for nuanced assessments of possible consequences.

Of course, it is highly unlikely that Blair would have changed his mind and refused to send British troops to Iraq if he had been given expert warnings that an occupation would meet serious resistance from Iraqi nationalists, that Islamists would fill the vacuum after Saddam was removed, and that Iraq would help al-Qaida find new recruits. He was set on going to war at Bush's side under any circumstances. But questions still need to be asked as to why the government's analysts did not do a better job of predicting the invasion's disastrous aftermath.

Blair's lack of interest in detail became clear to Britain's diplomats when he summoned key ambassadors back to London in January 2003 as war planning accelerated in Washington and arguments over a second Security Council resolution to authorise an invasion hotted up at the UN. In a lengthy speech, the prime minister outlined British policy on Iraq and the Middle East in general, naturally without conceding that a decision to invade had already been taken. He was telling the ambassadors how to sell the line rather than seeking their advice.

"Blair basically harangued us. I don't remember anyone giving any feedback," I was told by one ambassador who had come back from the Gulf for the occasion, and has since retired. The following day Mike O'Brien, the foreign office minister responsible for the region, held a smaller meeting with British heads of mission in the Middle East. "He told us they were trying to impose democracy in the Middle East. I said I didn't think it would work, but we were not asked for our advice and we didn't give any," the former ambassador recalled. "The issue was not posited in the context of, 'Should we invade or not?' "

Another British ambassador who attended the meetings and talked to colleagues about the looming invasion said: "Everyone was underprepared for the aftermath." He admitted that "to my shame I was in the complacent camp ... We underestimated the insurgency. I didn't hear anyone say, 'It'll be a disaster, and it'll all come unstuck.' People felt it was a leap in the dark, but not that we were staring disaster in the face."

The leader of the country where he served was far more perceptive about post-Saddam Iraq than the Foreign Office's Arabists. "He predicted it would all fall to pieces on sectarian grounds. He was unhappy about the invasion, even though he was a host to US forces and the top US brass came through regularly," the ambassador told me.

In London the Foreign Office set up a special Iraq policy unit in the run-up to the war. But it had a narrow brief, concentrating on contingency planning for the invasion and its short-term effects, according to a diplomat who attended its meetings. What would happen if Saddam's forces used chemical weapons and British forces took heavy casualties? The government had plans to commandeer hospitals in Britain's National Health Service, if army hospitals were swamped. What if hundreds of thousands of Iraqis fled the bombing? Plans were made for huge tented camps and emergency food supplies, to be run with the United Nations. The Department for Trade and Industry tried to guess what would happen to oil prices in the event of war. The Department for International Development focused on humanitarian assistance to refugees, and reconstruction. No discussions were held on vital issues such as how to choose Iraq's future government after Saddam fell, and what role the occupiers should play. Would Iraqis or the Americans be in charge?

According to Clare Short, the secretary of state for international department who resigned from the government shortly after the invasion, the cabinet had only informal prewar discussions on Iraq. "There were never any papers or proper analysis of the underlying dangers and the political, diplomatic and military options. The whole crisis was handled by Tony Blair and his entourage with considerable informality," she recorded later. Her worry was that without a UN resolution the occupiers would have no legal right to make political changes in Iraq. Peter Hain confirmed that the cabinet saw no papers on postwar Iraq. "In Iraq the failures of covert intelligence were compounded by the absence of political intelligence: a comprehensive lack of the understanding of sectarian forces and fault lines present across the country," he disclosed recently.

Blair was not interested in these matters. He took the view that it was in Britain's strategic interest to go along with whatever Bush decided. Civil servants and senior British military sources repeatedly complained that he never raised difficult problems with Bush, even when he had been briefed to mention them before going to Washington. He either lacked consideration for the consequences of an invasion, or perhaps he feared risking his friendship with Bush by sounding like a sceptic or a wimp. He thought he had considerable influence in the White House, and his various trips to Washington, which always culminated with a press conferences at Bush's side, were designed to give the impression that as a major contributor of troops he was an equal partner in decision-making.

British officials were under no such illusions. "We weren't plugged into the state department's detailed planning exercise. We tried but couldn't get into it. It was the first warning sign that we weren't part of it," one senior diplomat told me. In the words of another: "The UK supplied 10% of the invasion force. We provided 10% of the staff of the Coalition Provisional Authority. We had 10% of input into policy." In the final weeks before the invasion, the Pentagon wrested control of postwar planning away from the state department, leaving British ministers even more in the dark. A senior British officer was attached to US central command in Florida, but the main issues of Iraq's postwar governance were not discussed there. Even in Washington, among the neocons who were leading the drive for an invasion, there was no clear idea whether to appoint Iraqis to run the country or put a US overlord in charge. This was only decided after Saddam was toppled.

Unlike France, Germany and Italy, the British had no embassy in Baghdad in Saddam's final 12 years of rule. This left them bereft of good on-the-ground intelligence. It also meant there were few people in the Foreign Office with direct experience and knowledge of Iraq. As a result, the British did not predict the rise of Iraq's Islamists, whose strength destroyed the American neoconservative project for a liberal, secular and US-friendly democracy. "The conventional view was that Iraq was one of the most western-oriented of Arab states, with its British-educated, urban and secular professionals. I don't think anyone in London appreciated how far Islamism had gone, not just among the Shia, but the Sunnis, too," Christopher Segar, who took part in the prewar discussions and headed the British office in Baghdad after the invasion, told me.

Thanks partly to their Baghdad embassy, the French were better informed. They saw the potential for tensions between religious and secular forces in Iraq if Saddam were toppled. They also sensed that occupation would create resistance. "We believe that the use of force can arouse rancour and hatred, fuel a clash of identities, of cultures," Dominique de Villepin, the French foreign minister, declared in a speech to the UN Security Council two weeks before the invasion. For his part, president Jacques Chirac argued that the war would be perceived in the Arab and Muslim world as an attack on Islam. "A war of this kind cannot help giving a big lift to terrorism," he told Time magazine. "It would create a large number of little Bin Ladens."

The British government got almost everything wrong before the war. A senior Foreign Office official, who saw the few papers that were written about the invasion's likely consequences, told me: "The basic assumption that turned out to be false was that Iraqis felt themselves more Iraqi than Sunni or Shia." The papers also predicted that "in the south there would be a welcome and it would be less difficult than in Baghdad, where it would be harder to manage a transition". "We underestimated the difficulties. No one realised how difficult it would be," the official said.

British ambassadors in the region concentrated on telling London what sort of support for the invasion was likely to be given, publicly and privately, by the Arab governments to which they were accredited. The Gulf states and Jordan wanted Saddam removed. Syria did not. There was little analysis of what the "Arab street" would feel or what their official Arab contacts saw as the fallout in Iraq.

British diplomats at the UN also failed to warn London, either by not seeking their Arab colleagues' advice or not passing it on. In this they were less efficient than diplomats from the countries that were on the Security Council, but not as permanent veto-bearing members. Juan Gabriel Valdes, a former foreign minister of Chile who served as his country's UN envoy in 2003, represented one of the 10 countries that the British wooed hard for support for a second UN resolution. The British claimed to have intelligence about Saddam's efforts to cheat the UN weapons inspectors - evidence that the Chilean ambassador and his colleagues did not find convincing, even though they had no evidence of their own.

"The fact that we didn't have intelligence didn't mean that we didn't have good common sense," Valdes said later. He and his colleagues decided to talk to "every one of the members of the Arab group at the UN, but particularly with Jordan and specially with Saudi Arabia and other countries that were good friends of the US". They predicted, in private, "exactly what has happened historically in Iraq. It was not very difficult to get that information - that if the war happens, Iran would take an enormous role, that the situation would be absolutely catastrophic, and that the turn of events would leave the US and Great Britain involved in an atrocious situation."

A senior Foreign Office official admitted to worrying that Iran would benefit from the invasion more than other countries. "I remember saying to myself that we might be in a position of having destroyed Iraq and leaving a resurgent Iran," he told me. Typically, he never communicated his concern to ministers. His reason, he said, was that as the war drew nearer, the mood in Downing Street discouraged officials from raising problems.

· Jonathan Steele Extracted from Defeat: Why They Lost Iraq, by Jonathan Steele, published by IB Tauris at £20, and in the United States in March by Counterpoint. To order a copy for £18 with free UK p&p, go to theguardian.com/bookshop or call 0870 836 0875.