The UK's most secretive court is beginning a week-long hearing – mostly in public – into complaints that GCHQ's mass surveillance of the internet violates human rights.

The case against the monitoring agency at the investigatory powers tribunal (IPT) is the result of revelations by the US whistleblower Edward Snowden. It has been brought by Privacy International, Liberty, Amnesty International, the American Civil Liberties Union and a number of other overseas human rights groups.

The legal challenge is the first of dozens of GCHQ-related claims to be examined in detail by the IPT, which hears complaints against British intelligence agencies and government bodies that carry out surveillance under the Regulation of Investigatory Powers Act (Ripa).

The civil liberties organisations are concerned that their private communications may have been monitored under GCHQ's electronic surveillance programme Tempora, whose existence was revealed by Snowden. They also complain that information obtained through the US National Security Agency's Prism and Upstream programmes may have been shared with the British intelligence services, sidestepping protections provided by the UK legal system.

On 10 July the government announced emergency measures to preserve the legality of data retention and at the same time promised a review of how Ripa operates.

James Welch, Liberty's legal director, said: "As legislation is introduced to paper over one crack in the crumbling surveillance state, another faces challenge. Not content with forcing service providers to keep details of our calls and browsing histories, the government is fighting to retain the right to trawl through our communications with anyone outside and many inside the country. When will it learn that it is neither ethical nor efficient to turn everyone into suspects?"

Most IPT hearings are conducted behind closed doors. Since the tribunal was established 14 years ago, no complaint against the intelligence services has ever been upheld. There is no appeal against the court's decisions although the European court of human rights in Strasbourg has signalled that it will consider appeals from the IPT on the presumption that claimants have exhausted domestic remedies.

Mr Justice Burton, who became president of the court last October, describes it as an "open tribunal" and has vowed to make its procedures less clandestine. In a departure from previous practice, the IPT posted advance notice of this week's hearing on its website, explaining that the case is "against the intelligence agencies in respect of alleged interception activity involving UK and US access to communications".

In defence documents already submitted, the government's most senior security official, Charles Farr, has explained how searches on Google, Facebook, Twitter and YouTube, as well as emails to or from non-British citizens abroad, can be monitored legally by the security services without obtaining an individual warrant because they are deemed to be "external communications".

The IPT hearing, at the Rolls Building in central London, may hear some of the most sensitive evidence about interceptions in private. This claim is expected to concentrate on the legality of two interception programmes, Tempora and Prism, and their use by the UK's monitoring agency, GCHQ, and its US counterpart, the NSA.

In Farr's submission, he says he can "neither confirm or deny" the existence of Tempora, although he does acknowledge that Prism exists "because it has been expressly avowed by the executive branch of the US government".

Much of the tribunal's deliberations will therefore have to proceed on the basis of agreed hypothetical facts. For example, if Tempora exists, lawyers will ask, does it violate the rights to privacy and freedom of expression enshrined in articles 8 and 10 of the European convention on human rights?