A prominent Libyan dissident cannot pursue his "well-founded" claim that he was unlawfully abducted in a joint MI6-CIA operation, and later tortured, because to do so would damage Britain's relations with the US, a high court judge ruled on Friday.

The judge ruled that Abdel Hakim Belhaj could not sue MI6 and the former foreign secretary Jack Straw, even though he admitted that parliamentary oversight and police investigations were "not adequate substitutes" for a decision by a court of law.

Though the ruling, by Mr Justice Simon, dismissed Belhaj's case, it directly challenged the British government's argument that the parliamentary intelligence and security committee (ISC) was the proper body to investigate MI6 operations involving the rendition, detention, and alleged abuse of terror suspects.

Simon said he ruled against Belhaj because American, as well as British, officials were involved in the operation – the rendition of Belhaj and his pregnant wife to Tripoli in 2004 – which Belhaj wanted a British court to declare unlawful.

He concluded that to pursue the case might damage Britain's national interest – a reference to the government's claims that it would harm its relations with the US.

Simon said he came to the conclusion "with hesitation". Belhaj's lawyers intend to appeal against the judgment.

Simon said his hesitation arose from concern that "what appears to be a potentially well-founded claim that the UK authorities were directly implicated in the extraordinary rendition of the claimants will not be determined in any domestic court; and that parliamentary oversight and criminal investigations are not adequate substitutes for access to, and a decision by, the court".

He said the potential effect of the principle of "state doctrine", which bound him in this case, was "to preclude the right to a remedy against the potential misuse of executive power and in respect of breaches of fundamental rights".

The judge added that he had to decide that "the conduct of US officials acting outside the US was unlawful, in circumstances where there are no clear and incontrovertible standards for doing so and where there is incontestable evidence that such an inquiry would be damaging to the national interest".

Speaking from Tripoli, Belhaj, who is represented by the law firm Leigh Day and supported by the legal charity Reprieve, said: "The judge was obviously horrified by what happened to my wife and me. But he thought the law stopped him hearing our case because it might embarrass the Americans.

"I believe that the British justice system, which I admire greatly, is better than that. My wife and I will continue to seek the truth."

Sapna Malik, from Leigh Day, said: "If this judgment stands, it will mean that anything our security services do alongside the US government is immune from the British legal system – even MI6 officers arranging the rendition of a pregnant woman into the arms of Gaddafi. Our clients will seek to appeal."

Cori Crider of Reprieve said: "As the judge said himself, today's decision shuts out of court a man who was rendered to Gaddafi's torture chambers with his pregnant wife in 2004. The Libya rendition cases are the most ignominious episode of Britain's role in the 'war on terror' to date. The judge is also entirely right that the discredited intelligence and security committee is no substitute for British justice in a court of law."

Friday's decision came a day after the government decided to abandon its promised judge-led inquiry into the role of UK security and intelligence agencies and gave the task to the parliamentary ISC instead.

The government on Thursday released an interim report by the retired appeal court judge Sir Peter Gibson on MI5 and MI6 operations abroad. Gibson said he found "the United Kingdom may have been inappropriately involved in some renditions".