A secret court order that authorised a massive trawl by the National Security Agency of Americans' email and internet data was published for the first time on Monday night, among a trove of documents that also revealed a judge's concern that the NSA "continuously" and "systematically" violated the limits placed on the program.

The order by the Fisa court, almost certainly its first ruling on the controversial program and published only in heavily redacted form, shows that it granted permisson for the trawl in part beacause of the type of devices used for the surveillance. Even the judge approving the spying called it a “novel use” of government authorities.

Another later court order found that what it called "systemic overcollection" had taken place.

Transparency lawsuits brought by civil liberties groups compelled the US spy agencies on Monday night to shed new light on the highly controversial program, whose discontinuation in 2011 for unclear reasons was first reported by the Guardian based on leaks by the former NSA contractor Edward Snowden.

In a heavily redacted opinion Colleen Kollar-Kotelly, the former presiding judge of the Fisa court, placed legal weight on the methods of surveillance employed by the NSA, which had never before collected the internet data of “an enormous volume of communications”.

The methods, known as pen registers and trap-and-trace devices, record the incoming and outgoing routing information of communications – traditionally phone calls made between individual users. Kollar-Kotelly ruled that acquiring the metadata, and not the content, of email and internet usage in bulk was harmonious with the “purpose” of Congress and prior court rulings – even though no surveillance statute ever authorized it and top officials at the justice department and the FBI threatened to resign in 2004 over what they considered its dubious legality.

“The court recognizes that, by concluding that these definitions do not restrict the use of pen registers or trap-and-trace devices to communication facilities associated with individual users, it is finding that these definitions encompass an exceptionally broad form of collection,” wrote Kollar-Kotelly in an opinion whose date is redacted.

The type of data collected under the program included information on the "to", "from" and "bcc" lines of an email rather than the content. According to the government’s declaration to Kollar-Kotelly the NSA would keep the internet metadata “online” and available to analysts to search through for 18 months, after which it would be stored in an “‘offline’ tape system” available to relatively few officials. It would have to be destroyed four and a half years after initial collection.

Metadata, wrote Kollar-Kotelly, enjoyed no protection under the fourth amendment to the US constitution, a precedent established by the supreme court in 1979 in a single case on which the NSA relies currently.

Still, Kollar-Kotelly conceded that she was blessing “a novel use of statutory authorities for pen register/trap and trace surveillance”.

While at times Kollar-Kotelly appeared in her ruling to be hesitant about granting NSA broad authorities to collect Americans’ internet metadata, “deference”, she wrote, “should be given to the fully considered judgment of the executive branch in assessing and responding to national security threats and in determining the potential significance of intelligence-related information.”

James Comey testifies before the Senate judiciary committee during a hearing on his nomination to be FBI director. Photograph: Pete Marovich/Zuma Press/Corbis Photograph: Pete Marovich/Zuma Press/Corbis

The legal status of the internet metadata program was highly controversial. In March 2004 several justice department and FBI individuals threatened to resign – including James Comey, George W Bush’s deputy attorney general and now Barack Obama’s FBI director – if the Bush White House and NSA persisted in authorizing the program over their objections that the internet metadata bulk collection was insufficiently legally grounded.

An internal NSA draft history, first reported by the Guardian, noted that the program paused in March 2004, while the White House quelled the secret rebellion, but resumed in July after then NSA director Michael Hayden sought to reassure Kollar-Kotelly, who “signed the first” so-called Pen Register/Trap and Trace Order on 14 July 2004.

It is unclear if the order from Kollar-Kotelly released on Monday is her order of 14 July 2004 as the date is redacted.

Systemic overcollection

A later opinion on the internet metadata program, by Kollar-Kotelly’s successor, John Bates, states that the “NSA exceeded the scope of authorized acquisition continuously” after Kollar-Kotelly’s initial approval.

Bates wrote that subsequent NSA reporting to the court revealed that "systemic overcollection" had taken place from almost the beginning of the program. "Virtually every" record generated by the program "included some data that had not been authorized for collection". A footnote suggests that Kollar-Kotelly grew worried that the content of communications was collected at times despite her initial confidence that the collection methods of the pen registers and trap and trace devices could not permit that.

"The government has provided no comprehensive explanation of how so substantial an overcollection occurred," Bates wrote in another undated opinion, whose redactions suggest the NSA blamed "noncommunication with the technical personnel directly responsible".

A senior intelligence official, Shawn Turner of the Office of the Director of National Intelligence, told the Guardian in July that the Obama administration shut down the bulk internet metadata collection program in 2011 “for operational and resource reasons” and it had not been restarted.

Senator Ron Wyden, a member of the Senate intelligence committee, who has campaigned against the scope of NSA domestic surveillance, has suggested in several statements that the program was wasteful, violated Americans’ privacy and did not lead to useful counterterrorism information.

Hundreds of pages

The NSA also released hundreds of pages of documents on Monday related to training on use of its vast data troves; its certifications to Congress and the court about its bulk phone records collections on Americans; and its internal checks to prevent abuse.

The release was prompted by a lawsuit sponsored by the ACLU and the Electronic Frontier Foundation. Monday’s disclosure represented the final round of court-compelled document disclosures as part of the civil liberties organization’s attempts to learn more about what they contend is legally dubious mass surveillance on Americans phone records.

“On the logic of these opinions almost every digital footprint we leave behind can be vacuumed up by the government – who we talk to, what we read, where we go online," said Patrick Toomey, an attorney with the ACLU. "Like previous releases these materials show the danger of a government that sidesteps public debate and instead grounds its surveillance powers in the secret opinions of a secret court. The more we learn the clearer it is that our surveillance laws and oversight rules are in dramatic need of reform."

The release comes at the beginning of an important week in Washington for the NSA’s bulk phone records collection. On Thursday the NSA deputy director is scheduled to testify before a Senate panel that is considering a bill to strip the surveillance agency of its power to collect phone data from Americans without individual warrants. Legislators are also discussing attaching surveillance restrictions to an annual defence authorization bill that the Senate is taking up this week.

Monday was also a busy day for the NSA’s bulk surveillance in the courts. The supreme court declined to take a case about the bulk phone records collection, while a judge on a lower federal court considered an injunction against the NSA.

“I don't know, frankly, how I'm going to come out,” said Judge Richard Leon, who heard arguments for and against an injunction on the bulk phone records surveillance on Monday brought by the conservative group Judicial Watch in his US district court for the District of Columbia.

“It's going to the court of appeals and probably to the supreme court – one way or the other,” news organizations quoted Leon saying after the hearing.

At least one document related to bulk surveillance was not released.

The Fisa court announced on Monday afternoon that the intelligence agencies and the justice department decided that it would not declassify a court opinion from 19 February 2013 related to the court’s interpretation of Section 215. It is unclear if the opinion refers to bulk phone records collection or to the other sorts of records that the government contends the Patriot Act provision allows it to collect – such as financial data of the sort the Obama administration disclosed last week to the New York Times and Wall Street Journal that the CIA gathers.

The document disclosures came the same day that the US supreme court declined a request by the Electronic Frontier Foundation to review the legality of the bulk phone records collection.

A terse statement announcing several court orders did not address the reasons the court denied the review, brought by the Electronic Privacy Information Center, which argued that the Fisa court had no right to order telecoms to turn over customer data in bulk to the government.

Several other legal challenges to the bulk phone records collection are pending before lower federal courts. One of them, brought by the ACLU, will begin oral arguments on Friday in the Southern District of New York.

Marc Rotenberg, the executive director of Epic, said in a statement that he was “disappointed” in the supreme court’s decision as he called the bulk surveillance order “clearly illegal”.