Evidence that British intelligence agencies mounted their own "rendition" operation in collaboration with Muammar Gaddafi's security services has emerged with the discovery of a cache of Libyan government papers in an abandoned office building in Tripoli.

A secret CIA document found among the haul shows that the British and Libyans worked together to arrange for a terrorism suspect to be removed from Hong Kong to Tripoli – along with his wife and children – despite the risk that they would be tortured. The wording of the document suggests the CIA was not involved in the planning of the rendition operation, but was eager to become engaged during its execution and offered financial support.

Other papers found in the building suggest MI6 enjoyed a far closer working relationship with Gaddafi's intelligence agencies than has been publicly known, and was involved in a number of US-led operations that also resulted in Islamists being consigned to Gaddafi's prisons.

On Sunday, one of the victims, Abdul Hakim Belhaj – now commander of the anti-Gaddafi militia in Tripoli – demanded an apology from London and Washington and said he was considering suing over his rendition to Tripoli and subsequent torture.

For several years, senior MI5 and MI6 officers have sought to deny that their agencies have been guilty even of complicity in the rendition operations mounted by the US after 9/11, and the subsequent torture of the victims.

The discovery of the papers suggests that on one occasion, at least, the British ran their own "rendition to torture" operation. The victim was named by the CIA as Abu Munthir. He is thought to have been a man who used this nom de guerre while living in the UK, where he is said to have encouraged a group of British Muslims to mount a bomb attack on an unspecified target in the south-east of England. The plotters were under surveillance by MI5 and counterterrorism detectives at the time that Abu Munthir was detained in Hong Kong in March 2004 before being sent to Libya.

While five members of the gang were jailed for life after a trial at the Old Bailey, and a sixth received a 10-year sentence in Canada, the fate of Abu Munthir and his family remains unknown.

The papers were discovered by staff of Human Rights Watch, the New York-based NGO, in the unmarked offices of Libya's external security agency. A number of the documents detail meetings between the British and Libyans during the period of rapprochement that followed the 2003 invasion of Iraq, when Gaddafi was being persuaded to abandon his nuclear weapons programme.

The fact that MI6 and Libyan intelligence enjoyed a close relationship at this time is known: the Secret Intelligence Service made no secret of its role in the successful WMD negotiations, and when Gaddafi's former intelligence chief Moussa Koussa defected last March, MI6 organised the flight. The papers show that Sir Mark Allen, the former head of counterterrorism at MI6, played a key role in nurturing this relationship.

The documents also show that British intelligence agencies provided intelligence reports on individuals of interest to Tripoli, helped the Libyans identify at least one organisation using particular telephone numbers in the UK, and were intimately involved in a number of US operations that saw Islamist terrorist suspects rendered to Libya. Since the ousting of Gaddafi it has become apparent that the regime's enemies were tortured routinely while imprisoned, and at least one rendition victim, Ibn Sheikh al-Libi, later died in what the Libyans claimed was a suicide.

The CIA fax that details the UK-Libya rendition operation is potentially the most damning for the UK authorities, however. It was sent to Tripoli on 23 March 2003 and marked SECRET/US ONLY/EXCEPT LIBYA. "Our service has become aware that last weekend LIFG [Libyan Islamic Fighting Group] deputy Emir Abu Munthir and his spouse and children were being held in Hong Kong detention for immigration/passport violations," it says. "We are also aware that your service had been co-operating with the British to effect Abu Munthir's removal to Tripoli, and that you had an aircraft available for this purpose in the Maldives."

The fax goes on to explain that although Hong Kong had no wish to see a Libyan aircraft land on its territory, "to enable you to assume control of Abu Munthir and his family", the operation would work if the Libyans were to charter an aircraft registered in a third country, and that the US would assist with the cost. The Hong Kong authorities were also insisting that the Libyans offer an assurance that the family's human rights would be respected, but human rights groups would say that such assurances were worthless.

Whitehall officials on Sunday defended the actions of the intelligence agencies and their links with Libya, saying this was "ministerially authorised government policy". They said there were genuine fears some Libyan dissidents living in the UK posed a potential threat to national security, because of the group's links to Islamic extremists. They were cut in 2009.

MI5 and MI6 have continued to maintain they have not been complicit in torture and rendition despite the emergence of a growing body of evidence to the contrary. For example, the last Labour government tried, unsuccessfully, to prevent the high court disclosing evidence that MI5 knew Binyam Mohamed was being tortured in Pakistan before an officer was sent to interrogate Also, a secret telegram signed by Jack Straw while he was foreign secretary, which was disclosed in a second court case, showed that the government had decided a number of British nationals should be sent to Guantánamo Bay, but only after MI5 had interrogated them in Afghanistan.

Despite this, the agencies have continued to insist they were guilty only of being "slow to detect the emerging pattern" of rendition by the US, a defence that was accepted by the intelligence and security committee, the Westminster body that was established to offer political oversight of the agencies.

The secret CIA fax is the first sign that the British went much further than being merely complicit, and were directly involved in rendition to a country where the victim could expect to be tortured.

Abu Munthir was thought to be the link man between a group of British jihadists, whom he had met in Luton, and Abdul Hadi al-Iraqi, who has been accused of being a senior figure in al-Qaida.

A month after Abu Munthir's detention in Hong Kong and removal to Tripoli, 18 men were arrested in police raids across the south of England. Two other men were arrested in New York and Ottawa and several were seized in Pakistan. It was alleged at a trial at the Old Bailey that Abu Munthir had encouraged members of the group to mount attacks in the UK, rather than wage jihad in Afghanistan.

One of those held in Pakistan was Salahuddin Amin, then 29, from Luton, who was questioned 13 times by MI5 officers in between being tortured by agents of Pakistan's Inter-Services Intelligence (ISI). He was shown a photograph of Abu Munthir, and told that he too was in detention.The secret interrogation policy that MI5 and MI6 officers were instructed to follow during such operations was disclosed by the Guardian last month. Amin was later deported to the UK and is one of the men now serving life sentences.

The questioning of Amin by the ISI, under torture, appears to have been co-ordinated with the questioning of other suspects held by Scotland Yard at Paddington Green police station in west London. It now appears that it was also co-ordinated with the questioning – quite possibly also under torture – of Abu Munthir in Tripoli.