The Obama administration is invoking an obscure legal privilege to avoid judicial scrutiny of its secret collection of the communications of potentially millions of Americans.

Civil liberties lawyers trying to hold the administration to account through the courts for its surveillance of phone calls and emails of American citizens have been repeatedly stymied by the government's recourse to the "military and state secrets privilege". The precedent, rarely used but devastating in its legal impact, allows the government to claim that it cannot be submitted to judicial oversight because to do so it would have to compromise national security.

The government has cited the privilege in two active lawsuits being heard by a federal court in the northern district of California – Virginia v Barack Obama et al, and Carolyn Jewel v the National Security Agency. In both cases, the Obama administration has called for the cases to be dismissed on the grounds that the government's secret activities must remain secret.

The claim comes amid a billowing furore over US surveillance on the mass communications of Americans following disclosures by the Guardian of a massive NSA monitoring programme of Verizon phone records and internet communications.

The director of national intelligence, James Clapper, has written in court filings that "after careful and actual personal consideration of the matter, based upon my own knowledge and information obtained in the course of my official duties, I have determined that the disclosure of certain information would cause exceptionally grave damage to the national security of the United States. Thus, as to this information, I formally assert the state secrets privilege."

The use of the privilege has been personally approved by President Obama and several of the administration's most senior officials: in addition to Clapper, they include the director of the NSA Keith Alexander and Eric Holder, the attorney general. "The attorney general has personally reviewed and approved the government's privilege assertion in these cases," legal documents state.

In comments on Friday about the surveillance controversy, Obama insisted that the secret programmes were subjected "not only to congressional oversight but judicial oversight". He said federal judges were "looking over our shoulders".

But civil liberties lawyers say that the use of the privilege to shut down legal challenges was making a mockery of such "judicial oversight". Though classified information was shown to judges in camera, the citing of the precedent in the name of national security cowed judges into submission.

"The administration is saying that even if they are violating the constitution or committing a federal crime no court can stop them because it would compromise national security. That's a very dangerous argument," said Ilann Maazel, a lawyer with the New York-based Emery Celli firm who acts as lead counsel in the Shubert case.

"This has been legally frustrating and personally upsetting," Maazel added. "We have asked the government time after time what is the limit to the state secrets privilege, whether there's anything the government can't do and keep it secret, and every time the answer is: no."

Virginia Shubert, a housing expert from Brooklyn who is the first named plaintiff in the case, said she joined it because she considered the vast monitoring of telecommunications and emails in the wake of 9/11 to be an erosion of her rights. She called the use of the state secret privilege in blocking the action "absurd. When the government faces allegations that it has violated the constitution, it cannot hide behind state secrets to avoid accountability."

The Shubert lawsuit, first lodged with the courts in May 2006, alleges that the US government has operated a massive dragnet of private citizens' communications across the country. Drawing on the testimony of several whistleblowers, the suit accuses the Bush and then Obama administration of having broken the fourth amendment of the US constitution that guards against unwarranted searches and seizures by intercepting "en masse the communications of millions of ordinary Americans".

In the course of protracted legal argument the government has invoked the military and state secrets privilege no fewer than three times. The privilege was originally laid down in 1953 in a case in which the widows of Air Force personnel involved in a secret test run of a B-29 bomber that crashed sued to see a copy of the accident investigation report and were rebuffed under a claim of privilege that disclosure of the document would "expose military matters … in the interest of national security".

In court motions, the Obama administration has set out the information that it claims is exempt from legal scrutiny under the privilege, including "information that may tend to confirm or deny whether the plaintiffs have been subject to any alleged NSA intelligence activity" and "any information concerning NSA intelligence activities, sources, or methods that may relate to or be necessary to adjudicate plaintiffs' allegations."

The government goes further and says that the state secrets privilege also covers "allegations that the NSA, with the assistance of telecommunications carriers such as AT&T and Verizon, indiscriminately intercepts the content of communications and also collects the communication records of millions of Americans."

The second case, Jewel versus National Security Agency, was lodged in 2008 following the disclosures of an AT&T whistleblower, Mark Klein. He revealed in 2006 that the telecoms firm had set up a secret NSA room within its San Francisco office in which all phone calls from the region were passing through a splitter cabinet that sent a copy to the NSA.

Mark Rumold, an attorney with the Electronic Frontier Foundation working on Jewel, said that this week's disclosures by the Guardian would make it increasingly difficult for the administration to claim the state secrets privilege.

"The Guardian's disclosures may fundamentally alter the government's approach as they are going to have a tough time convincing a judge that this stuff is secret," he said.