The government's top foreign policy advisers were as inept as their US counterparts in failing to see that removing Saddam Hussein in 2003 was likely to lead to a nationalist insurgency by Sunnis and Shias and an Islamist government in Baghdad, run by allies of Iran, the Guardian has learned.

None of Whitehall's "Arabists" warned Tony Blair of the difficulties which have plagued the occupation. The revelation undermines the British claim that it was US myopia which was to blame for the failure to foresee what would happen in postwar Iraq.

"Everyone was unprepared for the aftermath," a former ambassador, who served in the region at the time, told the Guardian. "To my shame I was in the complacent camp [in the Foreign Office]. We underestimated the insurgency. I didn't hear anyone say, 'It'll be a disaster, and it'll all come unstuck'. People felt it was a leap in the dark but not that we were staring disaster in the face."

Privately, and in rare cases publicly, British ministers and officials have blamed the chaos of the occupation on blunders in Washington, pointing the finger particularly at Donald Rumsfeld, who was sacked as defence secretary in 2006. The Guardian's researches reveal that Britain's analysts were equally wrong.

Christopher Segar, who took part in Whitehall's Iraq Policy Unit's prewar discussions and later headed the British office in Baghdad immediately after the invasion, said: "The conventional view was that Iraq was one of the most Western-oriented of Arab states, with its British-educated, urban and secular professionals. I don't think anyone in London appreciated how far Islamism had gone."

Officials alone cannot be blamed. Ministers failed to ask serious questions. Blair never called on the experts for detailed analysis of the consequences of an invasion, officials say. He saw the war as Iraq's liberation and felt any postwar problems would pale in the face of Iraqi delight.

Opposition parties urged the government last year to authorise a full-scale independent inquiry into Whitehall's prewar discussions, but Blair refused to. Gordon Brown has taken the same line. The two men claim it would be wrong as long as British troops remain in Iraq.

The Conservatives will renew their calls for an inquiry in a House of Lords debate on Iraq on Thursday. The latest revelations are likely to increase pressure on the government. Lord Hurd, former foreign secretary, said last night: "Blair and his colleagues sent British troops to kill and be killed in Iraq without proper planning ... An inquiry is certainly needed to make sure this cannot happen again."

In the absence of a public inquiry, the Guardian interviewed a range of recently retired officials who now feel freer to talk about the crucial pre-invasion period.

Contrary to the conventional view that the occupation's problems stem mainly from failure to plan for postwar Iraq, they say there was plenty of planning, from how to react to mass refugee flows and a humanitarian crisis to the fallout from a sharp rise in the world price of oil. The real failure, they concede, was one of political analysis. Officials did not study how Iraqis would react to an occupation and what political forces would emerge on top once Saddam was removed.

One British diplomat who was based in the region and kept a special watch on the Shia Islamists admitted he did not foresee their postwar rise. "The issue of secularism versus religion was discussed but none of the leaders of Sciri seemed very strong," he said, referring to the Supreme Council for the Islamic Revolution in Iraq, a Shia grouping close to Iran. "I don't think anyone could have formed a view of the relative appeal of Sciri and [the other main Shia bloc] Dawa." He added that the maverick Shia cleric Moqtada al-Sadr, whose Mahdi army has constantly defied occupation forces "was unheard of". Yet in post-invasion Iraq the two Islamist groupings, Sciri and Dawa, became the biggest electoral parties. One reason for the weakness of British expertise was that, unlike France, Germany and Italy, Britain had not had an embassy in Baghdad since 1991, which meant fewer diplomats with direct knowledge of the country.

At the urging of Jack Straw, then foreign secretary, Blair held a brief meeting with six British academic specialists in November 2002. It was never repeated. Charles Tripp of the School of Oriental and African Studies, and author of the standard work, A History of Iraq, said that apart from this meeting in Downing Street, "I can't remember participating in any meaningful seminar on Iraq with the Foreign Office."