The UK's intelligence services have come under renewed pressure with the emergence of a fresh cache of secret documents that suggest MI5 officers forced Libyans seeking asylum in Britain to co-operate with the regime they had fled.

For the last three months, Scotland Yard detectives have been investigating MI6's alleged involvement in two so-called rendition operations that saw two Libyan dissidents kidnapped along with their families and flown to one of Muammar Gaddafi's prisons in 2004.

The role MI6 is said to have played is described in a batch of documents discovered in an abandoned government office in Tripoli last September. The two men have lodged civil claims against MI6 and against Jack Straw, who was foreign secretary at the time.

Well-placed officials said on Sunday that a key question, and the one that worried them, was how much information MI5 offered Libya about individuals in Britain. That issue is central to investigations now under way into MI5 and MI6 relations with Gaddafi and his security and intelligence agencies, they made clear.

The Mail on Sunday says it is in possession of a second batch of Libyan government documents, which, it claims, show how the continuing co-operation between the UK government and Gaddafi's external security operation, EOS, saw Libyan agents enter Britain in 2006 in order to make joint "approaches" to dissidents living in Manchester and London.

This meant "Britain conspired with Gaddafi's henchmen to menace his enemies on UK soil", the newspaper claims. The paper also says that in 2006 two Libyan agents were greeted at Heathrow and given a safe house in Knightsbridge and secure mobile phones.

The Mail claims that, according to the minutes of one meeting between MI5, MI6 and EOS officers in Tripoli, the MI5 officer said that "target 2 could become a very good source and we can pressure him to work for us because he's not a British citizen".

During another meeting, MI5 is reported in the paper to have warned its EOS counterparts: "The target person has the right to make a complaint or seek police protection. British intelligence must be careful how they approach a target because this individual could call on human rights or the press and cause a security scandal that exposes co-operation between British and Libyan secret services."

There was anger among lawyers who have represented Libyan asylum seekers, with one, Gareth Peirce, saying it had become common for MI5 to seek to exploit "vulnerable people who are facing deportation".

David Davis, the former shadow home secretary, said the 2004 "rendition" operations were not the only joint enterprises between the UK and Gaddafi's intelligence agency. "This is two years later. It involves people who have come to Britain, presumably cleared to live in Britain, being put under pressure to deal with the security service, which seems rather peculiar if this is realpolitik rather than protecting the security of the country."

Margaret Beckett, who was foreign secretary at the time, told the Press Association that the level of co-operation between Libya and the UK depicted in the files "didn't ring any bells with me".

One Libyan dissident, an accountant living in London, told the Mail he was shocked to discover that a photograph he had had taken when applying for a British passport in 2002 had been passed to the EOS and was among the recovered documents. He says he suspects that a telephone was being monitored, with the result that an associate in Libya was detained, tortured and then held for five years in one of Gaddafi's jails.

Whitehall sources do not deny that Libyan refugees and dissidents in Britain were watched by MI5. They say that this was the time the Libyan Islamic Fighting Group had links with al-Qaida, links which, they say, were not broken until 2009.

This is the explanation Whitehall officials give in defence of the security and intelligence agencies for the information they helped pass to Gaddafi about Libyans in the UK – and in trying to recruit them as informants. It is also the argument they give for their involvement in their rendition to Tripoli into the hands of Gaddafi's secret police.

The parliamentary intelligence and security committee (ISC) is investigating the activities of MI5 and MI6, but will have to await police inquiries.

Straw has refused to comment on reports that he personally authorised the 2004 rendition operations, and that he is to be interviewed by police.

Sir Malcolm Rifkind, the Conservative former foreign secretary and chairman of the ISC, said his committee had already heard some evidence about MI6 links with Gaddafi's intelligence service but its inquiry had to be suspended because of the police investigation. "We have every intention of resuming it as soon as the police investigation is over," he said.

Lord Goldsmith, Labour's former attorney general, said he was concerned by the allegations. He told the Murnaghan programme on Sky News: "I didn't know anything about this particular case and I'm very troubled by these allegations. We have obviously got to see what comes out … I think the issue of what was going on with Libya at that time, given what we subsequently know about Gaddafi, that's a fair thing to be looking into and I think we do need to look further into that."

Goldsmith said he understood why Tony Blair decided to travel to Libya in 2004 to meet Gaddafi after the then Libyan leader decided to abandon his weapons of mass destruction programme. "The problem was that Gaddafi did do something that was important in this period of actually accepting that he would get rid of his weapons of mass destruction," he said.

But Goldsmith added: "We thought, I think the world thought, he had turned for the good, and he hadn't. That was as it turned out to be a wrong judgment. Whether it was the wrong thing to do at the time, that's another issue, but this is a serious allegation and it obviously needs to be looked at."