Yet another Foreign Intelligence Surveillance Court (FISC) judge has blasted United States intelligence officials for disregarding the court’s guidelines for domestic surveillance of American e-mail metadata traffic, a program that ran for around a decade before ending in 2011.

“[National Security Agency’s] record of compliance with these rules has been poor,” wrote Judge John D. Bates in a 117-page opinion (PDF) whose date was redacted. The opinion is just one of a series of documents released and declassified late Monday evening by the Office of the Director of National Intelligence (ODNI).

“Most notably, NSA generally disregarded the special rules for disseminating United States person information outside of NSA until it was ordered to report such disseminations and certify to the FISC that the required approval had been approved. The government has provided no meaningful explanation why these violations occurred, but it seems likely that widespread ignorance of the rules was a contributing factor.”

This set of documents, which include annual reports from the Attorney General to Congress, memos, presentations, and training documents, were released in relation to an Electronic Frontier Foundation and American Civil Liberties Union lawsuit. Earlier batches were released in September 2013 in August 2013. In total, ODNI says it has now released nearly 2,000 new documents in recent months.

“Release of these documents reflects the Executive Branch’s continued commitment to making information about this intelligence collection program publicly available when appropriate and consistent with the national security of the United States,” James Clapper, the head of the ODNI, wrote on Monday.

“Additionally, they demonstrate the extent to which the Intelligence Community kept both Congress and the Foreign Intelligence Surveillance Court apprised of the status of the collection program under Section 215 [of the Patriot Act]. Some information has been redacted because these documents include discussion of matters that continue to be properly classified for national security reasons and the harm to national security would be great if disclosed.”

The Bates opinion is the second of the two most revealing documents in this new tranche. The first, written by FISC Judge Colleen Kollar-Kotelly, responds to a government request that NSA be allowed to use pen register and trap and trace devices (“pen/trap devices”) as a way to access metadata on electronic communication. She granted approval for the bulk surveillance but laid out specific guidelines for collection.

The second important FISC opinion, authored by Judge Bates, came in response to a government request that aimed to expand the metadata collection program by “11-24 times.” Bates slammed the government for not adhering to its guidelines but “reluctantly” allowed them to continue out of deference to the Executive Branch (and intelligence agencies, like the NSA, whose powers are granted through the Reagan-era Executive Order 12333). In the opinion, Judge Bates appears unwilling or unable to meaningfully punish any government officials despite clear violations of the court’s prior orders.

“I see a lot of similarities between the Bates opinion and the Walton opinion,” Mark Rumold, a staff attorney at the Electronic Frontier Foundation, told Ars. Rumold was referring to a 2009 opinion by FISC Judge Reggie Walton, who also lambasted the government for breaking the rules.

“It’s essentially the same thing, FISC taking NSA and [the Department of Justice] to task for violating their orders, for accessing more information than they were allowed to access under the orders and laying out under the ways that they had violated the court’s orders, [but then] letting them continue,” Rumold added. “The executive branch has pushed the judiciary so far and hopefully now we’re at that tipping point that the judiciary is comfortable with and they’ll start pushing back on executive misrepresentations.”

Not your father's pen/trap application

The Kollar-Kotelly opinion (PDF) described her response to a government application that “seeks authority for a much broader type of collection than other pen register/trap and trace applications,” compared to what had previously been done.

Pen/trap surveillance arises from a type of legal order that has recently skyrocketed in use in the US. Originally designed to apply to telephone companies, these orders are now being increasingly applied to tech companies as a way to capture user metadata, too. Of the total number of American law enforcement orders that it received in six months, Google said recently that two percent of those were pen/trap orders.

Applied to a Google user, for example, a pen register would likely record who that user was sending e-mail to. A corresponding “trap and trace order” would include metadata from e-mails received, likely including date, time, IP address, and other routing information. It could also include attachments and perhaps even—if broadly interpreted—anything but the actual content of an e-mail. Secure e-mail service Lavabit recently received such an order prior to its shutdown.

In the Monday night Tumblr post, the ODNI defined this now-shuttered program:

Under the now-discontinued [pen/trap] program, the FISC, after finding that the Government’s applications satisfied the requirements of FISA and the Constitution, approved orders that enabled the Government to collect electronic communications metadata, such as the “to,” “from,” and “cc” lines of an email and the email’s time and date. This program did not authorize the collection of the content of any electronic communications. Under this program, NSA could not read the content of any electronic communications for which the metadata was acquired. Like NSA’s bulk telephony metadata program, this program was subject to several restrictions approved by the FISC, such as: -The information had to be stored in secure databases.

-The information could be used only for counterterrorism purposes.

-The databases could be queried using an identifier such as an email address only when an analyst had a reasonable and articulable suspicion that the account or email address was associated with certain specified foreign terrorist organizations that were the subject of FBI counterterrorism investigations. The basis for that suspicion had to be documented in writing and approved by one of the 22 designated approving officials identified in the Court’s Order. Moreover, if an identifier was reasonably believed to be used by a United States person, NSA’s Office of General Counsel would also review the determination to ensure that suspected association was not based solely on First Amendment-protected activities.

-NSA was required to destroy the bulk metadata after a set period of time.

RAS-ma-tazz

It’s not known exactly how the government defined the parameters of the metadata it believed it could collect. The Kollar-Kotelly opinion (PDF) contains nearly three redacted pages outlining the “categories of information [the government] proposes to collect.” The judge notes—as many federal judges have—that metadata is not protected under the Fourth Amendment that shields Americans from unreasonable search and seizure. Why? Thanks to the legal rationale known as the “third-party doctrine.”

In 1976, the Supreme Court ruled in a landmark case (known as Smith v. Maryland) that when someone calls a telephone number, that number has been disclosed to a third party (the phone company). Therefore, the Supreme Court held, it is not private (because it was disclosed through the act of making the call), and the government can have easy access to those call records. The principle has been broadly extended to other kinds of metadata.

But what the US government was asking Kollar-Kotelly to approve was far more than just a pen/trap device on a single, or a small number of e-mail accounts. As she wrote:

In an effort both to identify unknown and to track known operatives [REDACTED] through their Internet communications, NSA seeks to acquire meta data, as described above, from all e-mail [REDACTED] are described in detail in the application and the [Director of the NSA] Declaration. In brief, they are [3 PAGES OF REDACTIONS]. The raw volume of the proposed collection is enormous. NSA estimates that this collection will encompass [REDACTED]. In absolute terms, the proposed surveillance “will result in the collection of meta data pertaining to [REDACTED] electronic communications, including meta data pertaining to communications of United States persons located within the United States who are not the subject of any FBI investigation.” Some proportion of these communications—less than half, but still a huge number in absolute terms—can be expected to be communications [REDACTED] who bear no relation to [REDACTED]. Through the proposed bulk collection, NSA would acquire an archive of meta data for large volumes of communications that, in NSA’s estimation, represent a relatively rich environment for finding [REDACTED] communications through later analysis. NSA asserts that more precisely targeted forms of collection against known accounts would tend to screen out the “unknowns” that NSA wants to discover, so that NSA needs bulk collection in order to identify unknown [REDACTED] communications.

Nearly 20 pages later, the judge continued:

In this case, senior responsible officials, whose judgement on these matters is entitled to deference, see pages 30-31 above, have articulated why they believe that bulk collection and archiving of meta data are necessary to identify and monitor [REDACTED] operatives whose Internet communications would otherwise go undetected in the huge streams of [REDACTED]. These officials have also explained why they seek to collect meta data [REDACTED] identified in the application. Based on these explanations, the proposed collection appears to be a reasonably effective means to this end.

Judge Kollar-Kotelly outlined very specific guidelines for the intelligence community to use this information, in particular what may be one of the earliest uses of the “reasonable articulable suspicion” (RAS) standard. As she concluded:

Such information shall be accessed only through queries using the contact chaining [REDACTED] methods described at page 43 above. Such queries shall be performed only on the basis of a particular known [REDACTED] after the NSA has concluded, based on the factual and practical considerations of everyday life on which reasonable and prudent persons act, that there are facts giving rise to a reasonable articulable suspicion that [REDACTED] is associated with [REDACTED] provided, however, that [REDACTED] believed to be used by a US person shall not be regarded as associated with [REDACTED] solely on the basis of activities that are protected by the First Amendment to the Constitution.

That RAS definition is more clearly outlined in a another document that was also declassified on Monday evening.

A document (PDF) dated August 29, 2008, a “Memorandum for the Deputy Program Manager for Counterterrorism Special Projects, Analysis and Production,” specifically illustrates this standard.