Five days after 9/11, early on a Sunday evening, a small group of senior CIA officers drove from their headquarters in Langley, Virginia, to the British embassy at 3100 Massachusetts Avenue NW in Washington DC, in order to brief MI6 on the agency’s planned response to the attacks.

How Britain did Gaddafi’s dirty work – podcast Read more

Leading the delegation was Cofer Black, head of the CIA’s counter-terrorist centre. Black was still wearing the same suit he had put on five days earlier, and looked shattered: he had been working day and night to draw up a cogent plan to protect his country from any further attacks.

Inside the embassy, Black and his colleagues gave a three-hour presentation on their plans. The CIA had been running a kidnap-and-interrogation project on a small scale since the mid-90s, targeting jihadists in Bosnia. It was known as the rendition programme. The plan was to dramatically increase the scale and scope of the programme.

According to Tyler Drumheller, then head of CIA operations in Europe, the MI6 officers listened quietly as Black detailed his plan, which involved the identification, abduction and interrogation of al-Qaida suspects around the globe. At the end of the presentation, Mark Allen, head of counter-terrorism at MI6, observed rather dryly that “it all sounds rather blood curdling”. Drumheller noted that the MI6 officers appeared very worried. Some of his fellow CIA officers, with less experience of dealing with the Brits, mistook their slightly insouciant manner for a sign of approval.

Allen wanted to know what the CIA and MI6 would do after al-Qaida was scattered across the world. He asked: “And what are we going to do, once we have hammered the mercury in Afghanistan?” The CIA officers looked at each other. According to one account of the meeting, Black said: “We’ll probably all be prosecuted.”

Facebook Twitter Pinterest Sir Mark Allen, former head of counter-terrorism at MI6. Photograph: Youtube

The death toll of the 9/11 attacks was still rising and President George W Bush was eager to take a tough stance. The day after the Pentagon briefing, Bush gave a press conference in which he offered a glimpse of what was to come. “I want justice,” he said. “There’s an old poster out west that says: ‘Wanted – Dead or Alive.’”

The implications of this would be thrashed out early the following year by the heads of the intelligence agencies of the US, UK, Canada, New Zealand and Australia. At a meeting at Queenstown, a ski resort in New Zealand, CIA director George Tenet insisted they could only understand and defeat al-Qaida if they worked closely with intelligence agencies across the Muslim world, and if they agreed to do whatever it took to hit back against terrorists. “The shackles, my friends, have been taken off,” Tenet is said to have declared.

One of the most pressing imperatives was to create close ties with the intelligence agencies of the Arab world. With this aim, in 2002 the CIA and MI6 began co-operating with the Libyan External Security Organisation (ESO), Col Muammar Gaddafi’s notorious overseas intelligence agency.

The stated agenda was to learn more about militant Islamism, but that would change the following year once Allen and his British political masters saw an opportunity to enter into negotiations with Gaddafi over his programme to develop weapons of mass destruction. Gaddafi had been trying to develop nuclear capability since the early 1970s, initially by trying to acquire Indian-made weapons, and then by attempting to gain access to uranium ore and enrichment technology.

From the late summer of 2003, as the war in Iraq began to go badly for the US and its allies, it became increasingly clear that Saddam Hussein’s weapons of mass destruction – the WMD programme that officially justified the invasion of Iraq – did not exist. But if Gaddafi could be persuaded to abandon his own nuclear plans, those who had pressed for war against Iraq could claim the invasion had been vindicated.

As the CIA and MI6 built relationships with Libya, the two agencies assisted Libyan spies in the kidnapping of Gaddafi’s enemies. Two leading figures in the Libyan opposition who had fled the country were kidnapped, one from Hong Kong, one from Thailand, and flown back to Tripoli along with their wives and children. Both men were tortured. MI6 gave their Libyan counterparts questions for the prisoners, who, under extreme duress, led them to other Libyan dissidents in exile.

Opponents of the Gaddafi regime who had been living legally in the UK for years were detained by British police, and the British government made a determined attempt to have them deported to Tripoli. Asylum seekers and British-Libyan nationals in Manchester and London were menaced by Gaddafi’s agents, who were invited into the UK and permitted to operate on the streets of Britain alongside MI5. British intelligence handed over details of the targets’ telephone calls to the ESO, and their relatives and friends in Libya were arrested and threatened.

Details of the dark arrangements made by the intelligence agencies of the US, UK and Libya have been gleaned through interviews with government officials and victims of rendition, British government documents disclosed under the Freedom of Information Act, and material that emerged during a lengthy Scotland Yard investigation and a number of civil trials. In large part, however, what follows is based on several extraordinary caches of secret British, American and Libyan intelligence documents that were discovered amid the chaos of the Libyan revolution in 2011, scattered around abandoned government offices, prisons and officials’ private residences. Many of the most intriguing documents were found by Libyan civilians and human rights activists in September that year inside ESO’s offices. Others came to light in various government outposts after Gaddafi was captured and killed the following month. All together, they amount to many thousands of pages.

These papers show that the post-9/11 rapprochement between the Gaddafi regime and the west – and Tony Blair’s government in particular – went far deeper than was previously known.

The most highly publicised result of the renewed dialogue with Libya was the dictator’s announcement that he was abandoning his WMD ambitions, both his nuclear and chemical and biological programmes. Another coup was the signing of multimillion dollar gas and oil exploration deals. Quietly, however, the relationship also bore a more bitter fruit: the kidnappings, detention and beatings carried out and assisted by the CIA and MI6.

These hitherto-secret documents offer a unique glimpse of a realpolitik that would be unimaginable had it not been detailed on one page after another. They show that, in their eagerness to get close to Gaddafi and influence the dictator’s future conduct, Britain’s intelligence agencies were prepared to commit serious human rights abuses on his behalf.

On 20 September 2001, four days after the CIA briefing on the revved-up rendition programme, MI6 spy chief Mark Allen was sitting face-to-face with a senior Libyan intelligence officer. It was well known to western intelligence agencies that Gaddafi was in a panic after 9/11. Libya’s record as a state sponsor of terrorism was established: Gaddafi’s government had been behind the 1988 bombing of Pan Am Flight 103 over the Scottish town of Lockerbie, killing 270 people. It was suspected of involvement in the 1986 bombing of a West Berlin nightclub frequented by US servicemen, and the downing the French UTA Flight 172 over Chad in 1989 with the loss of 170 lives. Furthermore, Gaddafi had made no secret about supplying arms to the IRA.

The dictator, aware of his reputation for supporting terror, had quickly condemned the al-Qaida attacks, but was far from certain this would be sufficient to save him from the wrath of the US. According to a cable from the US ambassador in Cairo, David Welch, Gaddafi had been trying to “call every Arab leader on his Rolodex”, begging them to set up a summit to discourage the US from launching an attack on his country. When he called King Abdullah of Jordan, he had sounded hysterical, Welch wrote.

Facebook Twitter Pinterest Muammar Gadaffi (left) with Moussa Koussa at the 21st Arab League leaders summit in Qatar in 2009. Photograph: Abaca/Press Association

Sitting across the table from Allen at this meeting was one of the few people capable of calming down the Brotherly Leader. Moussa Koussa had been close to Gaddafi for more than 20 years, and at the time of the meeting was head of the ESO, the organisation that was thought to have organised many of Libya’s terrorist attacks. Gaddafi trusted Koussa, and respected his advice.

Allen and Koussa had much in common: both were in their early 50s, well educated, precise in their manners and fairly devout in their respective faiths: Islam and Roman Catholicism. They were fluent in each other’s native tongues. In the face of the threat from al-Qaida, both were utterly pragmatic. The secret papers show that Koussa offered to provide Allen with information extracted from men held in Libyan prisons, and that both men agreed that their countries’ counter-terrorism experts “should meet to discuss the enemies we both face”.

In a follow-up fax sent two weeks later, Allen said he was “very interested in joint penetration operations” against an organisation called the Al-Jama’a al-Islamiyyah al-Muqatilah bi-Libya, or the Libyan Islamic Fighting Group (LIFG). Although some LIFG members had joined al-Qaida in Afghanistan, its leadership had for years rejected the overtures of Osama bin Laden, and remained focused on the overthrow of Gaddafi. Despite this, Allen clearly believed LIFG members could provide information about the threat from al-Qaida. In particular, he expressed interest in one of the LIFG’s military leaders, a 35-year-old veteran of the Soviet-Afghan war by the name of Abdel Hakim Belhaj.

On 13 November 2001, Bush signed a military order authorising the widespread use of rendition and torture. A few days later, ESO and MI6 met again. By now, the Libyans noted, MI6 were “determined to experiment with recruiting sources”.

Britain was wading into the war on terror. In mid-November, the Anti-terrorism, Crime and Security Act gave the home secretary the power to detain individuals without trial – and the intelligence agencies more power to target suspects.

By August 2002, relations between Britain and Libya were tentatively restored, with the goal of what Blair would later call the “huge prize” of security co-operation with Libyan intelligence. Gaddafi’s son Saif was admitted to the London School of Economics at his second attempt. During a preliminary telephone conversation, Blair and Gaddafi exchanged pleasantries. Junior foreign office minister Mike O’Brien visited Gaddafi at Sirte, the dictator’s birthplace, and delivered a letter from Blair. It was the first British ministerial visit since 1984, when the UK had severed diplomatic relations after a police officer, Yvonne Fletcher, was murdered by shots fired from inside the Libyan embassy in London.

No matter that Gaddafi was regarded across the Middle East as dangerously insane. He may be a madman, those in the higher reaches of the British government appear to have concluded, but at least he’s our madman.

Much of the work involved in building the partnership was done not by ministers, or even by diplomats, but by intelligence officers, particularly Allen and Koussa. They met frequently over the course of 2002 and appear to have become firm friends. Gifts of dates and oranges from Tripoli began to appear at MI6’s headquarters on the south bank of the Thames, while on 20 September 2003, on the first anniversary of the two men’s initial meeting, Koussa was invited to “a banquet dinner at the Goring”, a luxury hotel.

It was Koussa’s first visit to the UK in decades. In June 1980, as head of the Libyan People’s Bureau, as his country’s embassy in London was then known, he had given an extraordinary interview to the Times in which he admitted having given his personal approval for the murder of two Libyans resident in the UK. Lord Carrington, then the foreign secretary, had barely permitted Koussa’s feet to touch the ground as he was bundled out of the country.

And here he was, 23 years later, being wined and dined a stone’s throw from Buckingham Palace. “The head of the British delegation warmly welcomed his guest, expressing his happiness,” according to the Libyan minutes. There does not appear to have been any discussion about the murders Koussa had authorised in London two decades earlier. Nor was there any mention of the 10 bomb attacks that were aimed at Gaddafi’s opponents in Manchester and London in March 1984, injuring 29.

Instead, there were talks about “bilateral and international issues”, intelligence sharing, and about MI6’s desire that Saif Gaddafi should be safe and comfortable while living in London. The conversation then turned to the dictator’s political opponents living in the UK. “What we need to deal with these individuals who reside in the UK,” a senior MI6 officer is reported as saying, “is tangible evidence to take instant actions against them.”

Emboldened by the new spirit of co-operation, in October 2002 Blair wrote to Gaddafi, suggesting that the sanctions that were holding back his country’s oil industry and placing an enormous burden on its economy might be lifted, if he would agree to abandon his WMD programme, which had long been a matter of great concern to the west.

On 25 November 2002, the Libyans passed MI6 a list of 79 Libyan opposition activists, who they referred to as “heretics”, living in the UK. Most, if not all, were said to be members or supporters of the LIFG.

An underground group formed in 1995 by Libyans who had fought against the Soviet Union in Afghanistan, the LIFG was dedicated to the overthrow of Gaddafi and the establishment of an Islamic government in Tripoli. It had announced its existence through a series of clashes with Gaddafi’s forces in the east of the country. For many years it had two leaders: Belhaj was its military commander, while another man, Sami al-Saadi, was its spiritual leader.

While the LIFG leadership never condoned al-Qaida’s attacks on the west, and insisted it was concerned only with the overthrow of Gaddafi, hundreds of its members joined al-Qaida in Afghanistan following a failed assassination attempt on the dictator in 1996, and after 9/11, the US government listed it as a terrorist organisation. It was not proscribed in the UK at this time, however. Many of its members had fled to Britain, as well as to China and Iran. Al-Saadi, his wife, Karima, and four children were among those who settled in London briefly, before moving to Tehran.

Facebook Twitter Pinterest Abdel Hakim Belhaj instructs anti-Gaddafi troops in Triploi in 2011. Photograph: Getty

In the UK, the LIFG was tolerated by the government. They were able to regroup and raise funds. From late 2002, however, as the rapprochement between London and Tripoli warmed up, Libyans resident in the UK were stopped and questioned at airports, and there were police raids on family homes in London and Manchester. When the UK-based Libyan author Hisham Matar – whose father, a noted member of a different dissident group, had been disappeared by the Gaddafi regime in 1990 – dined out in London, he began choosing seats that faced the door of the restaurant. “None of us felt safe,” he later wrote.

In early 2003, as US and UK forces mustered on the borders of Iraq, Gaddafi was afraid they would target Libya. According to one diplomat, he called the Italian prime minister, Silvio Berlusconi, begging him to “tell them I will do whatever they want”.

In March, Blair secured the backing of the Commons for war against Iraq, and two days later the invasion began. Three weeks after that, Baghdad appeared to have fallen to the US and the war was thought to be all but over: on 1 May, on board the aircraft carrier USS Abraham Lincoln, Bush gave an address in front of a large banner declaring: “Mission Accomplished.”

The secret papers show that by then Allen was regularly meeting with a senior CIA officer, Stephen Kappes, to discuss ways they could make sure Gaddafi abandoned his ambitions to develop nuclear, chemical and biological weaponry. During their conversations with Koussa, no sanctions were threatened; there was little need, as all three men knew Gaddafi was terrified of being invaded by the US. At meetings in Tripoli, MI6 officers discussed not only the WMD programme, but ways of targeting Libyan “heretics” around the world. British intelligence officers told the Libyans they had intercepted Sami al-Saadi’s telephone calls from his home in Tehran.

At one meeting between the ESO, MI6 and MI5, the British passed over a briefing paper that MI5 had prepared. “Greetings from the British Security Service,” it read. “We … wish to share with you information that we have that may be of interest.” It contained details about the whereabouts and movements of Gaddafi’s opponents in London, Brighton, Peshawar and Los Angeles. MI5 also passed on details of “UK-based Libyan extremists”. British intelligence was starting to track the LIFG leadership. ESO asked the British if they could help capture Belhaj, who was in China with his Moroccan wife, Fatima Bouchar. MI6 replied that they must first sound out the Chinese.

British intelligence officers were not unaware of how this activity would be viewed at home. In advance of another meeting, Sadegh Krema, the deputy head of the ESO, passed around an internal note in which he warned that the British were particularly anxious that the meeting should remain “confidential”, because the “domestic political and legal situation [in Britain] is complicated”.

In Iraq, meanwhile, it was dawning on the US and its allies that the mission was very far from being accomplished. On 7 August, a car bomb exploded outside the Jordanian embassy in Baghdad, killing 17 and injuring dozens. Twelve days later, a massive bomb packed into a cement lorry blew apart the United Nations HQ at the Canal Hotel in Baghdad, killing UN special representative Sergio Vieira de Mello and 21 others. A determined insurgency was underway.

Confidence in the case for going to war was evaporating in London and Washington. The head of the Iraq Survey Group, a multinational body set up to find Saddam’s WMDs, announced that there was little evidence he possessed any.

If the US and UK could successfully disarm Gaddafi – and link that success to the war in Iraq – the invasion would not appear to have been such a ruinous miscalculation. Then, on the Mediterranean, north of Libya, the allies got a lucky break.

The BBC China, a German-registered cargo ship, left the Suez Canal on 4 October 2003 and sailed west towards Libya. Italian naval vessels intercepted the ship and forced her into the port at Taranto, where a search established that five shipping containers labelled “used machine parts” were packed with thousands of centrifuge components for Gaddafi’s uranium enrichment programme.

The components had been tracked from Malaysia, where they had been manufactured on behalf of Abdul Qadeer Khan, the Pakistani nuclear scientist who is credited with fathering not only his own country’s nuclear bomb, but also with providing the expertise and equipment that became the seeds for the North Korean and Iranian nuclear programmes. Libya had also turned to Khan’s one-stop proliferation shop.

Allen invited Koussa back to the UK. This time it was for a 90-minute meeting on 20 November at the Bay Tree, a wisteria-clad five-star hotel in the Cotswolds, just 10 minutes’ drive from the RAF base at Brize Norton. The Libyan minutes of the meeting explain that Allen and Kappes passed on personal messages for Gaddafi from both Blair and Bush, before coming to the point: they knew the Libyans were pressing ahead with their nuclear weapons programme, while pretending to dismantle it.

Facebook Twitter Pinterest Moussa Koussa (left) arriving at the foreign ministry in Madrid in 2010. Photograph: Pierre-Philippe Marcou/AFP/Getty

Koussa responded by pledging that the Libyan government would now scrap the programme and accepted that at this stage, “there is no room for evasion or dribbling or twisting”.

Eleven days later, a 13-strong US/UK inspection team, led by Kappes and Allen, flew into Mitiga airport in Tripoli. According to subsequent news reports in the US, they discovered evidence of a chemical weapons programme that was rudimentary and untested, and a nuclear weapons project that was at a surprisingly advanced stage. Work began to dismantle the dictator’s weapons plants and remove blueprints, centrifuges and other equipment from the country. Around 13kg of 80%-enriched uranium were reported to have been taken away with the assistance of Russia. The whole process would be completed in just four months.

It was a coup for Washington and Whitehall, who were determined to make the most of its propaganda value. Back in London, at the Travellers Club in Pall Mall, Allen met Robert Joseph, senior counter-proliferation official at the US National Security Council, to discuss the wording of a statement that Gaddafi was to make, announcing that he had abandoned his WMD ambitions.

It was agreed that Gaddafi would make the announcement on Libyan television on 19 December. Blair and Bush would then make their own statements. Capricious as ever, Gaddafi decided at the last minute that he wasn’t going to do it – he wanted to watch football on TV, apparently – and his foreign minister made the announcement instead.

In their responses, neither Bush nor Blair mentioned Iraq, but they did allude to the invasion. “We have shown resolve,” said Bush. “In word and in deed, we have clarified the choices left to potential adversaries.” Blair added that “recent events and political determination” had, after all, made the world a safer place. Opinion pieces written by supporters of the Iraq war claimed that disarming Gaddafi showed the war was justified.

On Christmas Eve, Allen sent a letter to Koussa, to be carried personally by the ESO courier who had just arrived in London, bearing dates and oranges. “It has been a real privilege working with you,” he wrote. “At this time sacred to peace, I offer you my admiration and every congratulation.”

But the suspension of Gaddafi’s WMD ambitions was not, it turned out, the end of their collaboration.

In February 2004 the LIFG commander Abdel Hakim Belhaj and his wife, Fatima Bouchar, who was four-and-a-half months pregnant, attempted to board a commercial flight from Beijing to London, where he hoped to claim asylum. Instead, the Chinese authorities deported the couple to Malaysia. On arrival in Kuala Lumpur they were detained.

Among the secret papers discovered in Koussa’s own office was a fax from MI6, dated 1 March 2004, which informed the Libyans of the couple’s location, and copies of letters from Koussa to the Malaysian ambassador to Libya, requesting his assistance. There is also a fax from the CIA, dated 6 March 2004, about “the capture and rendition” of Belhaj. “We are planning to take control of the pair in Bangkok and place them on our aircraft for a flight to your country,” it says.

The following evening, the Malaysian authorities put the couple on a commercial flight to London, via Bangkok. In Bangkok, they were taken off the aircraft, hooded, and taken to a CIA detention centre somewhere within Don Mueang international airport.

Facebook Twitter Pinterest Sami al-Saadi, an opponent of Gaddafi, in 2011. Photograph: Marco Longari/AFP

Belhaj says he was beaten and hung from hooks, and blasted by loud music. Bouchar told me that when she was dragged away from her husband, she feared he was going to be killed. “They took me into a cell, and they chained my left wrist to the wall and both my ankles to the floor. I could sit down but I couldn’t move.” Bouchar was chained to the wall for five days, and given water but no food. “They knew I was pregnant. It was obvious.” She was forced to lie on a stretcher, and was bound to it, head to foot, with sticky tape. They put a hood and earmuffs on her. She was unable to move, hear or see. “My left eye was closed when the tape was applied. But my right eye was open. It was agony.”

After five days, the pair were taken to Tripoli on a flight that took 17 hours. On arrival they were driven separately to Tajoura prison, east of the city. Belhaj says Koussa greeted him in person. Then he was chained to a wall, he says, and beaten. Almost immediately, MI5 and MI6 began sending questions that they wanted the Libyans to put to him. Many concerned the lives of other Libyan dissidents around the world. And the questions would keep coming. They are all there, in the secret papers discovered in Tripoli: more than 1,600 questions.

Nine days after the couple’s arrival at Tajoura, Allen sent to his friend Koussa what is perhaps the most remarkable fax of the entire series. He started by expressing his gratitude to his friend for arranging a visit by Tony Blair to Gaddafi – or “the Leader” as he put it. Blair, he explained, would be travelling with journalists, and would like to meet the Leader in his tent. “The plain fact is the journalists would love it,” Allen wrote.

He then congratulated Koussa on the “safe arrival” of Belhaj. “This was the least we could do for you and for Libya to demonstrate the remarkable relationship we have built over recent years.” He added that “amusingly”, the CIA had asked that MI6 channel all requests for information from Belhaj through them. “I have no intention of doing any such thing. The intelligence … was British. I know I did not pay for the air cargo. But I feel I have the right to deal with you direct on this.”

Blair visited Libya for the first time on 25 March 2004, six days after Allen sent his “safe arrival” fax. After being photographed shaking hands with Gaddafi, he announced that Libya had “found a common cause, with us, in the fight against extremism and terrorism”. At the same time, in London, it was announced that the Anglo-Dutch oil giant Shell had signed a £110m deal for gas exploration rights off the Libyan coast.

Two days after that, a second UK-US-Libyan rendition operation was underway. The LIFG spiritual leader Sami al-Saadi had also moved to China with Karima and the four children. The family had travelled to Hong Kong after al-Saadi approached MI5 via an intermediary to ask if they would be allowed to return to London. He was under the impression he was to be interviewed by British diplomats in Hong Kong. Instead, the entire family was detained by Hong Kong immigration authorities.

Al-Saadi’s eldest child, Khadija, then aged 12, recalls how she and her younger brothers, Mostapha, 11, and Anes, nine, and six-year-old sister Arowa, were separated from their parents before they were all taken aboard an Egyptian airliner, which was empty but for a small number of Libyan men.

“After a while I was allowed to go into the next compartment and see my mother,” she says. “She was crying. She told me they were taking us to Libya. Initially, I didn’t believe it. Then I realised it was true, and I was very scared. I thought we would all be killed. Then I was told to go and say goodbye to my father. He was handcuffed to a seat in another compartment and had a drip in his arm. I fainted.”

After the aircraft landed in Tripoli, Khadija watched as her parents were taken off and hooded, and then their legs bound with wire. Mostapha and Anes were blindfolded. The family was then driven in a convoy of vehicles to the prison at Tajoura.

Al-Saadi says he was beaten, threatened and subjected to electric shocks. Khadija knew her father was being tortured: every few days he was brought to see his family for a few minutes before being taken away again. “I think they were doing it to increase the pressure on him.” The children decided to embark on a hunger strike, “but they didn’t care whether we ate anything or not”. The children were released, along with their mother, after 10 weeks, and allowed to enrol in school. Al-Saadi, like Belhaj, would spend six years in Gaddafi’s prison.

Later that summer, Allen left MI6 and was appointed as an advisor to BP. At the end of the year he was knighted.

In 2005, the LIFG was banned in the UK. Three members were prosecuted for providing it with funds and false passports, and sentenced to between 22 and 45 months in prison. A number of other LIFG supporters in the UK were detained on the basis of information they say was extracted under torture from Belhaj and Al-Saadi. They were subjected to British control orders – detained pending deportation to Libya. Some of them had been granted asylum from the Gaddafi regime, and had been living peacefully in the UK for years. But now, it seemed, control orders were being used as instruments of international diplomacy.

The UK government signed a memorandum of understanding with the Gaddafi regime under which the Libyans pledged not to execute, torture or mistreat anyone who was forcibly returned. At one point, the British government suggested – in all seriousness – that Libyan compliance with the Memorandum could be monitored by a body called the Gaddafi Foundation, which was run by the dictator’s son, Saif. Such was Libya’s appalling human rights record, however – with opponents of Gaddafi facing murder, torture and arbitrary detention – that the government was never able to deport its detainees: in one judgment after another, the courts ruled that anyone returned to that country would have no chance of a fair trial.

In April 2007, Blair wrote a personal letter to Gaddafi. “Dear Mu’ammar,” he began, “I trust that you, and your family, are well. With regret, I should let you know that the British Government has not been successful in its recent Court case here involving deportation to Libya.” Blair thanked Gaddafi personally for the help he had given the British government in its attempt to secure deportation, and for the “excellent co-operation” between the two countries’ intelligence agencies. “Best wishes,” he signed off, “yours ever, Tony.”

Chastened, perhaps, by the courts’ verdict, the British side was anxious that their joint operations with Libya should never be made public. MI5 warned that steps should be taken jointly to “avoid being trapped in any sort of legal problem [and] to avoid also that those joint plans be discovered by lawyers or human rights organisations and the media”.

Meanwhile, in Tripoli, Belhaj and Al-Saadi were interrogated by two British intelligence officers. On one occasion, when left alone with his British visitors, Belhaj says he indicated that they were being covertly recorded, displayed the scars on his arms, and indicated through sign language that he was being suspended by his arms and beaten. The British clearly understood, he says: one gave a thumbs-up sign, while the other nodded her head.

Not that British intelligence believed the detention and interrogation of these men had made the UK – or indeed the world – a safer place. Among the papers discovered during the revolution was an MI5 memorandum prepared in advance of a visit to Tripoli in February 2005 marked “Secret, UK/Libya Eyes Only”, which contains a remarkably candid assessment. The detention of Belhaj and Al-Saadi had resulted in the LIFG being “cast into a state of disarray”, the memo states, before adding that these men had always jealously guarded the group’s independence from the worldwide jihadist movement, but in their absence, it was now falling under the influence of others who may be pushing it towards al-Qaida’s agenda.

Tony Blair’s “Dear Mu’ammar” letter was one of the first of the secret papers that came to light when the Libyan revolution exploded four years later. On 16 February 2011, inspired by the revolutions in Tunisia and Egypt, demonstrators gathered in the eastern Libyan town of Bayda, calling for the overthrow of Gaddafi. A police car was attacked and set ablaze at a road junction that is now known as The Crossroads of the Spark. It took more than eight months of vicious fighting, however, before the rebels, aided by Nato, toppled the regime. Belhaj played a role in the revolution, and was appointed leader of the militia that took control of the capital.

On 20 October, wounded and surrounded by rebels in Sirte, Gaddafi hid inside a drainage pipe. Grainy, jerking footage shot on a mobile phone captured the last moments of the despot whose friendship the UK had done so much to cultivate. He was dragged out, beaten, stabbed in the backside, and died. Three of Gaddafi’s sons died during the revolution, while Saif, captured by rebels, was abandoned by his friends in London and spent six years in jail.

Facebook Twitter Pinterest Documents showing co-operation between the CIA and Muammar Gadaffi’s intelligence agencies found at the offices of Moussa Koussa. Photograph: Sipa Press/Rex

One senior servant of the regime survived unscathed. Moussa Koussa had left the country after telling Gaddafi he needed medical treatment in Tunisia, where he boarded a private jet and flew to the UK. He immediately met with MI6 officers. Shortly afterwards, he left Britain, and settled in Qatar.

By then, the first secret cache of Tripoli intelligence documents had already been discovered, and some of their contents reported. With the role that MI6 played in the kidnap of Belhaj, Al-Saadi and their families now laid bare, Sir Richard Dearlove, the agency’s chief at the time, made a rare public statement in defence of MI6’s conduct. Government ministers had approved their actions, he said: “It was a political decision, having very significantly disarmed Libya, for the government to co-operate with Libya on Islamist terrorism.”

As a consequence of the discovery of the papers, Belhaj, Al-Saadi and several of the men who were detained and subject to control orders in the UK, or had their assets frozen on the basis of information being extracted from Gaddafi’s prisoners, brought damages claims against the British government and MI6 for the mistreatment they had suffered.

Al-Saadi settled his claim in 2012 when Britain paid him £2.23m in compensation, without admitting liability. Belhaj brought proceedings against not only the British government, but also against Allen and Jack Straw, who had been foreign secretary, and responsible for MI6, at the time that the agency assisted with his kidnap. Belhaj said he would settle for just £3 – £1 each from the government, Allen and Straw – providing both he and his wife received an unreserved apology. That was unlikely to happen, however: Scotland Yard had embarked upon a criminal investigation of the intelligence agency’s role in the Libyan rendition operations, and to make an admission of liability would be to invite arrests.

The investigation, codenamed Operation Lydd, ran for more than three years. Straw, who has always denied being complicit in unlawful rendition or detention, was questioned as a witness. Allen, who has declined to comment publicly, was the suspect. He faced potential charges of aiding, abetting, counselling or procuring kidnap, false imprisonment, assault or torture, and misconduct in public office. The police report contained evidence that Allen had been in contact with Koussa about the two rendition operations. However, in June last year, the CPS decided that there was insufficient evidence to bring charges against Allen.

Britain’s secret wars | Ian Cobain Read more

The Libyan renditions were also examined during a short-lived inquiry into the UK’s involvement in the mistreatment of terrorism suspects after 9/11. It had been set up by the coalition government following the 2010 UK general election, but was shut down once the police investigation began. Instead, its work was handed to Westminster’s Intelligence and Security Committee. Four years later, the committee has yet to report. Belhaj’s British lawyers are currently seeking a judicial review of the CPS decision to not prosecute Allen, and the couple’s damages claim is yet to be heard in the high court. Government lawyers fought for years to have the claim struck out, finally losing at the supreme court earlier this year. A hearing is expected next year.

In 2011, Dame Eliza Manningham-Buller, who was director general of MI5 from 2002 until 2007, asked “whether the UK supped with a sufficiently long spoon” in dealing with the Libyan dictator. Indeed, the story of Britain’s attempt to make an ally of Libya has an uncomfortable postscript. At a remote depot in the south east of Libya, the National Transitional Council that was established after the death of Gaddafi discovered that the dictator had concealed large stocks of chemical weapons. They found mustard-gas artillery shells and precursors for the creation of other chemical weapons. Gaddafi, it turns out, had never really shut down his weapons programme. The UK and the US had done a deal with one of the worst dictators of the last century, and he had fooled them.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.