The extent of the intelligence services' bulk interception of online communications came under scrutiny for the first time in a British courtroom on Friday.

Lawyers for MI5, MI6 and GCHQ faced challenges brought by nearly a dozen British and international civil liberties groups over the legality of US and UK digital surveillance programmes, including Tempora, Prism and Upstream.

Claims that the mass collection, storage and analysis of emails and electronic messages are illegal were made at the investigatory powers tribunal (IPT), which adjudicates on complaints against the intelligence services and surveillance by government bodies.

The government, adopting a "neither confirm nor deny" approach, is responding to allegations about the programmes on a hypothetical premise. The case follows a series of reports published in the Guardian last year based on revelations by the former US National Security Agency contractor Edward Snowden.

The IPT is already considering a complaint that the intelligence services illegally intercepted communications between the Libyan Abdel Hakim Belhadj – the subject of removal by rendition back to Tripoli – and his lawyers. Another complaint about mass surveillance is being taken directly to the European court of human rights in Strasbourg by Big Brother Watch, the Open Rights Group and English PEN.

The complaints to the IPT about bulk interception of online communication have been brought by Liberty, Privacy International, Bytes For All and the American Civil Liberties Union. Amnesty International has also joined the case. Other groups represented include the Canadian Civil Liberties Association, Egyptian Initiative for Personal Rights, Hungarian Civil Liberties Union, Irish Council for Civil Liberties and the Legal Resources Centre.

The full hearing, scheduled to last a week, will take place in July. Part of it is likely to be heard behind closed doors on the grounds of national security.

In written submissions to the preliminary hearing, James Eadie QC,for the government, said: "The [intelligence services] adopt a 'neither confirm nor deny' stance as to whether relevant steps [surveillance] were in fact taken in relation to the claimants, whilst accepting that the impugned regimes might in principle have been used."

Some of the hearings would have to be held in closed session, Eadie argued. "The true value of any covert intelligence-gathering measures can only be assessed by reference, at least in part, to sensitive intelligence matters that cannot be publicly revealed."

The claims submitted by the human rights groups describe Prism and Upstream as US-run mass data collection programmes. Tempora, said to be run by GCHQ, intercepts information and telephone information passing through fibre-optic cables.

"There is no clear legal framework, within the Regulation of Investigatory Powers Act 2000 or otherwise, that permits the vast collection and storage of communications carried out by Tempora. Such activity is not in accordance with law," Liberty argued in its written submission.

Amnesty International alleges that its communications have been intercepted illegally in breach of its rights to privacy and freedom of expression.

Privacy International said : "All internet and telephone communications, without meaningful limits, are being collected, stored and analysed by the security and intelligence services, regardless of any grounds for suspicion. This raises important issues of law and principle."

Last year the Guardian reported that GCHQ had secretly gained access to the network of cables that carry the world's phone calls and internet traffic and was processing vast streams of sensitive personal information, which it was sharing with its American partner the NSA.

The president of the IPT, Mr Justice Burton, said the tribunal was "unique on being able to hear cases on the basis of assumed facts". Burton made one interim ruling: that Tempora should be pronounced with a short 'o' rather than a longer vowel sound, to avoid confusion with Japanese deep fried food.

The case continues.