The treatment of prisoners at Guantanamo Bay and Abu Ghraib by the US authorities has been “hugely detrimental” to the fight against terrorism, “creating long-lasting grievances” against the West, the official historian of MI5 said today.

Christopher Andrew, whose authorised book on the history of the Secret Service has just been published, said: “There may be worse policies to have, but I can’t think of many." Use of torture, he added, was not only morally wrong, but created the danger of getting answers from suspects which were incorrect.

Allegations that MI5 agents had condoned the torture of Binyam Mohamed al-Habashi, a British resident, by foreign intelligence agencies, is the subject of an ongoing investigation by Scotland Yard.

Download the new Independent Premium app Sharing the full story, not just the headlines

Dr Andrew stressed that his trawl of thousands of classified documents did not reveal any instances of MI5 allowing torture by its operatives. However, he pointed out that a former Director-General, Dame Eliza Manningham-Buller, has acknowledged that “It is pretty well impractical always to check whether something has been derived from torture unless you have reason to suspect it at the beginning. Literally thousands of pieces of intelligence are shared daily between the UK, our allies and people who might not be so reasonably be described as our allies.”

In his book, The Defence Of The Realm, Dr Andrew stated that there had been scepticism within MI5 about George W Bush’s call for a "War on Terror". An in-house review lampooned it as "The War on Terry (WOT)". And the then-chief, Sir Stephen Lander, had to reassure his staff that although a military response was being considered, “You should be reassured that that political, humanitarian, intelligence and law enforcement responses are also high on the UK agenda.”

Dr Andrew stated that MI5 had been slow in recognising the threat posed by Muslim extremists. “A wide-ranging study on ‘The Origins of Terrorism’ commissioned in 1994 still saw no serious threat from the transnational Islamist terrorism which was to preoccupy the Service in the first decade of the 21st Century.”

In 1995 there were intelligence reports that Osama Bin Laden was involved in a suicide bombing plot in the UK. However, the fundamentalist threat was still not being seen as a priority when Dr Andrew raised the issue during a meeting at Mi5’s headquarters in January 1998.

Dr Andrew maintained that the failure to take early notice of the Islamist threat was due to "Historical Attention Span Deficit" - looking at short-term measures rather than an adequate study of history which would have shown its murderous emergence.

But the former MI5 chief Sir Stephen told The Independent yesterday: “We did have a full time job in our hand at the time with the Provisional IRA. If the evidence was there [of the Islamist threat] we would have acted on it.”

The Government, too, had ignored the rising jihadist tide, but, in the aftermath of 9/11 Sir Stephen said he found ministers and civil servants “were seeing suicide bombers around every corner… We had to calm people down and focus on the job at hand.”

On the issue of prisoner mistreatment, Sir Stephen said: “When we sent officers to speak to prisoners at Guantanama Bay I said to the Home Secretary that we’ll be accused of colluding in illegal detention. This is what happened. But it was still the right thing to do.”