David Cameron today ordered an unprecedented inquiry into evidence and allegations of British complicity in the torture and abuse of terror suspects.

But he immediately moved to ensure the courts would no longer be able to disclose damning evidence which, he implied, could jeopardise intelligence sharing with the US.

Honouring a promise while in opposition that he would set up a judge-led inquiry into mounting evidence, emerging mainly from court hearings, the prime minister told the Commons he had asked Sir Peter Gibson – a former appeal court judge who privately monitors the activities of the intelligence agencies – to "look at whether Britain was implicated in the improper treatment of detainees held by other countries that may have occurred in the aftermath of 9/11".

He said that while there was no evidence that any British officer was "directly engaged in torture" in the aftermath of 9/11 there were "questions over the degree to which British officers were working with foreign security services who were treating detainees in ways they should not have done".

Though he did not point directly to a particular case, he made clear he was referring to evidence disclosed by the high court that MI5 knew about the abuse of Binyam Mohamed, a British resident held incognito in Pakistan in 2002 before being secretly rendered to jails in Morocco, Afghanistan, and Guantánamo Bay.

In rulings highly critical of government officials, a string of senior judges ruled this year that CIA information showing MI5 involvement in abuse should be disclosed. The US made it clear it was appalled and suggested it would cut off intelligence sharing with the UK. To prevent any repeat of such disclosures, Cameron told MPs the government intended to publish a green paper setting out "proposals for how intelligence is treated in the full range of judicial proceedings, including addressing the concerns of our allies".

Government officials made clear ministers are seeking legislation that would in future prevent judges release information passed to MI5 by the CIA or by any other foreign intelligence agency.

He said the government wanted to pursue "mediation" with six former Guantánamo Bay detainees who had brought civil claims about their treatment – and who are demanding the disclosure of MI5 and MI6 intelligence. They will be offered out-of-court compensation.

The inquiry by Gibson, who will be assisted by Dame Janet Paraskeva, head of the civil service commissioners, and Peter Riddell, the former Times journalist and fellow of the Institute of Government, will not be able to summon foreign witnesses or compel former Labour ministers to testify.

However, the inquiry has the freedom to investigate the policies that underpinned MI5 and MI6 conduct. Critically, it could allow the inquiry to identify the government ministers who authorised those policies.

Widespread support for the inquiry was expressed , qualified by concern over the degree of secrecy that will surround it, and over its inability to compel former ministers to appear.

There was also concern that the government's new interrogation guidelines for British intelligence officers – published after being recently rewritten – contained a number of loopholes that could still lead to it being used to facilitate torture.

Andrew Tyrie, a backbench Tory MP who has been campaigning for an inquiry for several years, said: "The prime minister's courageous decision to hold a judge-led inquiry, and to set aside a number of objections that will have been put to him, is a huge step forward. It can give the public confidence that the truth on British involvement in rendition will come out."

Shami Chakrabarti, director of Liberty, said: "An inquiry into British complicity in torture is welcome and overdue but this announcement leaves room for fears that government is bending towards the security establishment.

"They wouldn't be in this mess but for all the excuses for secret stitch-ups instead of open justice. This inquiry can only be credible with the broadest remit, the most public proceedings possible and by full engagement with victims, witnesses and lawyers. Any attempt to exempt intelligence from legal scrutiny is an attempt to exempt the security services from the rule of law."

• This article was amended on 7 July 2010. Due to an editing change, the original attributed the statement from Liberty's Shami Chakrabarti to Andrew Tyrie MP. This has been corrected.