GCHQ is probably intercepting legally privileged communications between lawyers and their clients, according to a detailed claim filed on behalf of eight Libyans involved in politically sensitive compensation battles with the UK.

The accusation has been lodged with Britain's most secret court, the investigatory powers tribunal (IPT), which examines complaints about the intelligence services and government use of covert surveillance. Most of its hearings are in private.

The allegation has emerged in the wake of the Guardian's revelations about extensive monitoring by GCHQ of the internet and telephone calls, chiefly through its Tempora programme.

The system taps directly into fibre optic cables carrying the bulk of online exchanges transiting the UK and enables intelligence officials to screen vast quantities of data.

The eight Libyans, members of two families now living in the country's capital, Tripoli, say they were victims of rendition. They claim they were kidnapped by MI6 and US intelligence agencies, forcibly returned to Colonel Muammar Gaddafi's regime and tortured. At that time, in 2004, when Gaddafi relinquished his nuclear weapons programme, intelligence relations between Tripoli, London and Washington were close.

A landmark legal action between Abdel Belhaj, 47, and the UK government is due to be heard at the high court shortly to resolve the kidnap and torture allegations.

But lawyers working with the human rights group Reprieve fear their ability to fight the case will be undermined because their legal correspondence may be surreptitiously monitored.

Sami al-Saadi, another Libyan dissident, and his family have already settled their claim against the government for a payment of £2.2m. The Foreign Office did not, however, admit liability.

The "notice of complaint" by solicitors at Leigh Day on behalf of Reprieve and the Libyans has already been made to the IPT. It lists the Security Service (MI5), the Secret Intelligence Service (MI6), the Government Communications Headquarters (GCHQ) in Cheltenham, the home secretary and the foreign secretary as respondents. It calls for the case to be heard in open court.

The claims states: "There is a strong likelihood that the respondents have intercepted and are intercepting the applicants' legally privileged communications in respect of the [cases]." Belhaj and Saadi were prominent military leaders of the Libyan Islamic Fighting Group (LIFG) during the Libyan revolution, the document points out, and are therefore "likely to be of interest" to UK intelligence agencies.

Additionally, the complaint maintains, GCHQ's capability has enabled it to engage in "mass scale communication gathering as part of the tempora programme" that "monitors and collates, on a blanket basis, the full range of electronic communications...". Information obtained is routinely stored for three days but held for longer for those "deemed to be of interest".

The complaint quotes as evidence accounts published in the Guardian about the Tempora system based on documents revealed by the US National Security Agency (NSA) whistleblower Edward Snowden as part of his attempt to expose what he has called "the largest programme of suspicionless surveillance in human history".

'Legal professional privilege', as it is formally known, allows legal advice between clients and their lawyers to be kept private. According to the Law Society, which represents solicitors in England and Wales, it ensures that "certain documents and information provided to lawyers cannot be disclosed at all. It recognises the client's fundamental human right to be candid with his legal adviser, without fear of later disclosure to his prejudice. It is an absolute right and cannot be overidden by any other interest."

Reprieve and Leigh Day allege the surveillance breaches the European convention on human rights, which guarantees respect for private and family life and the right to a fair trial.

The only legal exemption for intercepting emails, the submission argues, would be for the prevention of crime – a consideration not relevant in this context. It also questions whether the Libyans have been discriminated against by being subjected to surveillance that would not have been imposed on UK nationals.

The Libyans' lawyers are seeking from the government a declaration that intercepting their privileged communications was unlawful, an injunction preventing interception and "just satisfaction" for breach of their human rights.

The complaint includes articles from the Guardian detailing the workings of the Tempora programme and the text of a fax sent by Sir Mark Allen, who was head of counter-terrorism at MI6 in 2004, to Mousa Koussa, then head of the Libyan external security organisation. It was recovered from Koussa's office following the revolution that toppled Gaddaffi.

The message read: "I congratulate you on the safe arrival of [Belhaj].

This was the least we could do for you and for Libya to demonstrate the remarkable relationship we have built up over recent years."

Belhaj was founder and leader of the LIFG which aimed to overthrow Gaddafi. He fled Libya in 1998 following an unsuccessful uprising and moved to China where he met his Moroccan wife, Fatima Boudchar. He was deported from China to Malaysia in 2004. From there he was forcibly transferred to Libya.

Saadi and his Algerian wife, Karima Ait Baaziz, and family had also moved to China where they were eventually detained in Hong Kong.

The CIA and British authorities are alleged to have conspired to return them to Tripoli on a charter flight.

Saadi said he was repeatedly tortured in a Libyan prison, being assaulted with fists, sticks, whips and rubber hosing. He claims he was interrogated by British and Libyan officials. Belhaj, who is now a politician in Libya, also maintains that he was tortured after being returned to Tripoli.

Cori Cryder, who heads Reprieve's legal team on abuses in counter-terroism, said: "It is bad enough that UK security services helped kidnap and render young children and a pregnant woman into the hands of Colonel Gaddafi.

"To add insult to injury, they are now trying to undermine their right to a fair trial by spying on private communications with their lawyers. UK complicity in Gaddafi's torture of his opponents is a shameful incident that needs to be opened up to public scrutiny – not subject to more skulduggery from GCHQ." The Home Office declined to comment last night on the grounds that the legal action between Belhaj and the government was continuing.