At the time of the 9/11 attacks on New York and Washington, Osama bin Laden was in an Afghan cave, unable to get a decent TV satellite signal and forced to follow developments on the radio. The contrast between his situation and his impact was to be a theme of the next decade until, eventually, the Americans caught up with him in the raid on Abbottabad, Pakistan, in May 2011. It’s a decade that The Exile describes with a remarkable amount of impressive new detail.

Investigative reporters Cathy Scott-Clark and Adrian Levy start with a detailed account of Bin Laden’s movements. When the US air strikes began, he flitted through various locations in Afghanistan, all the while trying to manage the movements of his wives and children. He was on his way to a meeting with Mullah Omar in Kandahar on 7 October 2001 when a US drone came close to killing them both. From there he moved to an underground complex in the Tora Bora mountains near the Pakistan border. The US assaulted Tora Bora but, again, Bin Laden managed to slip away, and on 14 December 2001 he turned up in the Pakistani city of Karachi. Feeling too exposed there, he moved back to Afghanistan in February 2002 before reaching northern Pakistan in the summer. There he lived with one of his wives, Amal, and their nine-month-old daughter Safiyah in the remote village of Kutkey, home to the in-laws of his courier and guard, Abu Ahmad al-Kuwaiti.

Bin Laden felt vulnerable, especially after 28 February 2003 when the main architect of the 9/11 attacks, Khalid Sheikh Mohammed, was arrested in Rawalpindi shortly after he had been to Kutkey. Kuwaiti, fearing Khalid Sheikh Mohammed would be forced to talk, scrambled to move Bin Laden to an uninhabited building owned by his father near Kohat in north-west Pakistan. It was an unsatisfactory solution and by 2004 al-Qaida, keen to settle and protect its leader, had managed to get a more suitable home built for him in the military town of Abbottabad. In August 2004 Bin Laden, together with a growing retinue of wives and children, moved in.

The main question that arose after the Abbottabad raid concerned the two most powerful men in Pakistan, general Ashfaq Kayani, the chief of army staff, and general Ahmad Shuja Pasha, the director-general of the main intelligence agency, the ISI. Had they known Bin Laden had been hiding in Pakistan? A lot hung on the answer. Many people in Washington believed Pakistan was not just an unreliable ally, but a thoroughly duplicitous one. Firm evidence that it had given sanctuary to the architect of 9/11 would make the country the ultimate state sponsor of terrorism.

There were reasons to think senior Pakistanis did know. First, it was difficult to believe that Bin Laden could have been living within a couple of miles of Pakistan’s military academy without the army being aware of his presence. Second, Pakistan had a track record of protecting and sponsoring violent jihadists and lying about it. And finally, the whole story seemed consistent with the ISI’s modus operandi. For years Pakistan had averted crises in its relationship with Washington by conveniently arresting an al-Qaida leader just as some senior American was heading towards Islamabad to read the riot act. The ISI liked to hold onto its cards until they could be played for maximum benefit. Bin Laden, the strongest card of all, would be played only when Islamabad was in dire need. Consequently, many writers and journalists who followed Pakistan closely and many Pakistanis, too, were predisposed to think that the ISI and the army probably knew.

Facebook Twitter Pinterest Osama bin Laden: how did he stay hidden for so long? Photograph: Uncredited/AP

But as time passed little evidence emerged to confirm that suspicion. Many journalists tried to follow up leads. There was a house in Abbottabad next to Bin Laden’s with a brass plate and the name of a major on it. Was he the minder? He denied it. There were the architectural drawings that had been submitted for the planning permission needed to build the house. The architects were rumoured to have ISI connections. Had the state built a safe house? The architects denied it. Denials may not amount to much but there were other indications pointing to a lack of state complicity. Credible reports from Washington said that the CIA had been listening to the phone calls of the army and ISI chiefs in the immediate aftermath of the attack and concluded they were genuinely surprised by what happened.

Relying on published documents and, more important, interviews with al-Qaida associates and Bin Laden family members, Scott-Clark and Levy try to answer the question of how much Pakistan knew. The Exile concludes that, while generals Kayani and Pasha were unaware, a former ISI chief did know about the Abbottabad house. General Hamid Gul, who died in 2015, was for many years one of Islamabad’s most prominent Islamists. He was director-general of the ISI from 1987 to 1989. Spotting him for the extremist he was, Benazir Bhutto quickly removed him when she became prime minister. Gul bitterly resented her decision but remained a highly active and prominent figure, who became a focal point for violent jihadists. Scott-Clark and Levy believe that Gul not only knew where Bin Laden was but also protected the al-Qaida leader by using his influence over the ISI’s “S” wing, which, they say, “is well known to be practically independent of and ungovernable by the ISI leadership”.

The overlaps between the ISI and violent jihadist outfits are so numerous and opaque that it is difficult to know where one ends and the others begin. Some militants are on the state payroll but maintain a degree of independence. Some retired ISI officers join militant groups but from time to time report back to their old employer. And yet the claim that some serving ISI officials took independent decisions on such a sensitive matter as Bin Laden’s location is highly contentious. The ISI may be far less competent than it is often portrayed. It might be so obsessed with secrecy that one part of the organisation does not know what another is doing. Notwithstanding Scott-Clark and Levy’s assertions, many ISI watchers will continue to believe that the chain of command remains intact and that, had a group in the lower ranks been protecting Bin Laden, the senior leadership would have known about it.

The Exile also tells the fascinating story of al-Qaida in Iran. For years now, snippets of information have emerged about how Iran protected some of Bin Laden’s senior associates and close relatives after 9/11. Many treated these claims with great scepticism. Was it really plausible, they wondered, that Iran would offer sanctuary to Shia-hating, violent Sunni jihadists? Scott-Clark and Levy provide a mass of new detail about what happened. It is a tale of missed opportunities. Straight after 9/11, officials in both Washington and Tehran realised that the attack might enable the two countries to start cooperating against their common enemies: al-Qaida and the Taliban. As early as 15 September 2001, a senior US state department official, Ryan Crocker, and the Iranian deputy foreign affairs minister, Mohammad Javad Zarif, started a series of face-to-face meetings in Geneva during which Iran shared intelligence about Taliban and al-Qaida positions in Afghanistan. But the reluctance of hardliners in Washington to accept Iranian help meant the efforts at rapprochement spluttered. And when, in January 2002, president Bush gave his “axis of evil” speech, which targeted Iran, they stalled altogether.

One of Scott-Clark and Levy’s best new sources is a Mauritanian cleric called Mahfouz Ibn al-Waleed. For more than a decade before 9/11, Mahfouz chaired al-Qaida’s sharia committee, pronouncing on the religious validity of the organisation’s policies and actions. He was one of the few people with advance knowledge of the 9/11 attacks. In fact his opposition to what al-Qaida insiders called the “planes operation” was so strong that he resigned from al-Qaida over it. Realising that this stand against Bin Laden would not be enough to protect him from American ire, Mahfouz, like other violent jihadists who had fled Afghanistan in the days after 9/11, needed somewhere to lie low. The situation was so desperate that they started talking about the possibility of going to Iran. Mahfouz made a hair-raising dash across the border to explore what might be possible. Sensitive to the different political trends in Tehran, he avoided contact with the Iranian government and dealt solely with the Revolutionary Guard. With the promise of immunity from al-Qaida attacks, Iran indicated it was receptive and by March 2002 there was a steady flow of senior al-Qaida figures and Bin Laden relatives moving into Iran.

Facebook Twitter Pinterest Policemen guard the compound in Abbottabad where Osama bin Laden was killed by US special forces. Photograph: Faisal Mahmood/Reuters

The Exile’s revelations lay bare the degree of animosity and distrust between the factions in Iran. When pro-government personnel in the ministry of intelligence and security tracked phone calls from the al-Qaida leaders in Iran to their affiliates left behind in Pakistan and Afghanistan, they started making arrests. The Iranian government even went as far as sending some al-Qaida members back to their countries of origin. Outmanoeuvred by these decisions, the Revolutionary Guard was left imploring the al-Qaida leaders not to use their phones.

Eventually Tehran’s disputatious leaders worked out a common position. How best to describe their compromise remains the subject of debate. The US argues that Iran harboured terrorists. Iran prefers to say that it detained them or, perhaps, put them in “protective custody”. The more senior al-Qaida leaders were housed within a training facility of the elite Quds Force in north Tehran. Some of Bin Laden’s relatives were put in a separate facility in the same compound. Others were put in safe houses and another group ended up in a provincial prison where conditions were so bad they went on hunger strike. None was free to leave Iran – unless they were going to fight US forces in Iraq.

Iran figured that the detainees were not only a good source of intelligence but also a bargaining chip. But with the Revolutionary Guard and US vice-president Dick Cheney both viscerally opposed to even mutually advantageous interaction, it was a deal that could never be done. The matter came to a head shortly after the invasion of Iraq in March 2003. With the US anxious about al-Qaida activity in Iraq, Crocker, now accompanied by Zalmay Khalilzad, president Bush’s special envoy to Afghanistan, approached Iran once again. And this time the Iranian government made a remarkable offer: if the US would give them the leaders of Mujahedeen-e-Khalq – an obscure Iraq-based cult that opposes the Iranian government – Tehran would hand over al-Qaida’s military council and Bin Laden’s family. There could be no greater demonstration of the depth of Washington’s hostility to Iran than the decision to turn down that opportunity.

It may be that some of the arguments in The Exile can be contested and that a few details turn out to be wrong. To give one example, the version of how Bin Laden’s son Hamza managed to flee Pakistan after the Abbottabad raid may need to be revised as new information comes to light. Scott-Clark and Levy write that al-Qaida managed to get him on a plane to Qatar, whereas others think his movements were facilitated by the ISI. But such quibbles should not take away from the authors’ achievement of speaking to previously silent senior al-Qaida figures and Bin Laden family members, not to mention just about all the key players inside Pakistan’s military establishment. They have produced the best account yet of what happened to al-Qaida after 9/11: it is an astonishingly good piece of work.

• The Exile: The Flight of Osama bin Laden is published by Bloomsbury. To order a copy for £21.25 (RRP £25) go to ?utm_source=editoriallink&utm_medium=merch&utm_campaign=articlebookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.

Owen Bennett-Jones’s Pakistan: Eye of the Storm is published by Yale.