Lawyers for Britain's intelligence agencies have given an undertaking that they would not read any intercepted communications between a pair of Libyan rendition victims and their British legal team, but declined to say whether emails and telephone conversations had been examined in the past.

The assurance was given at a public hearing of the UK's most secretive court, the Investigatory Powers Tribunal (IPT), which examines complaints about surveillance by the country's intelligence agencies and other public bodies.

The IPT usually sits in secret, and it remains unclear after Thursday's rare open hearing whether it would meet again behind closed doors to consider the complaint brought by a Libyan dissident, Abdel Hakim Belhaj, and his wife, Fatima Bouchar.

James Eadie QC, counsel for the government, GCHQ, MI6 and MI5, said he could not guarantee secret hearings would not take place. Initially he insisted that if such hearings did take place, the government would not explain to the tribunal why it was sitting behind closed doors, a position that he reversed after complaints from the couples' lawyers.

The couple were kidnapped in Thailand in 2004, bound and gagged and flown to Tripoli, where the heavily pregnant Bouchar was imprisoned for several months and her husband endured several years of imprisonment and torture.

Signed papers discovered by chance during the 2011 Libyan revolution showed that Sir Mark Allen, then head of counter-terrorism at MI6, wrote to the Gaddafi regime to ask it to acknowledge the key role his agency had played in the affair.

Belhaj and Bouchar are suing the government, Allen, and Jack Straw, who, as foreign secretary at the time, was the minister responsible for MI6. Most of their claim has been struck out after a high court judge ruled that although it was "well founded", it would be too damaging to UK relations with the US to allow it to proceed. The couple plan to appeal.

Following the disclosure by Edward Snowden that GCHQ is capturing details of most – if not all – email and telephone communications into and out of the UK, the couple demanded an assurance that no intelligence officers or Whitehall lawyers involved in contesting their claim would be permitted to see any communications with their lawyers that have already been captured.

The president of the tribunal, Mr Justice Burton, pointed out that warrants are issued to permit the UK's intelligence agencies to intercept privileged communications between lawyers and their clients.

While refusing to confirm or deny that communications between the couple and their lawyers had been intercepted, Eadie said the agencies were prepared to give assurances that any material that has been captured would not be seen by anyone preparing a defence for the high court claim.

The tribunal is also being asked to consider the extent of the secrecy that surrounds its work.

Last December the tribunal and government lawyers held a secret hearing to discuss the Libyan couple's claim. Dinah Rose QC said that some of the matters discussed – such as a government request for further time – should have been heard in public, and to discuss them in a closed session was "disproportionate and unlawful" and in breach of the couple's rights under European law to a fair trial.

Rose said the tribunal should give notice to complainants when it was planning to sit; should consider whether it should sit in open court; and the extent to which it needed to sit behind closed doors.

Although 15 government lawyers, civil servants and intelligence officers attended the hearing, Eadie said more time was needed to prepare the government's response. For the first time, the tribunal did give advance notice of a public hearing. Mr Justice Burton has said he will look at the tribunal procedures to see what can be done to make it more open in the way it conducts its judicial business.