More than a decade after the 9/11 terrorist attacks, a set of extraordinary and secretive surveillance programs conducted by the National Security Agency has been institutionalized, and they have grown.

These special programs are conducted under the code name Ragtime, and are divided into several subcomponents, according to the new book Deep State: Inside the Government Secrecy Industry, by Marc Ambinder and D.B. Grady. (I purchased a copy this morning.)

The authors, both journalists who cowrote a previous book about special operations in the military, have dug deep into the code names and operational nitty gritty of the NSA’s secretive and hugely controversial surveillance programs, and they’ve come up with impressive new details.

Ragtime, which appears in official reports by the abbreviation RT, consists of four parts.

Ragtime-A involves US-based interception of all foreign-to-foreign counterterrorism-related data;

Ragtime-B deals with data from foreign governments that transits through the US;

Ragtime-C deals with counterproliferation actvities;

and then there’s Ragtime-P, which will probably be of greatest interest to those who continue to demand more information from the NSA about what it does in the United States.

P stands for Patriot Act. Ragtime-P is the remnant of the original President’s Surveillance Program, the name given to so-called “warrantless wiretapping” activities after 9/11, in which one end of a phone call or an e-mail terminated inside the United States. That collection has since been brought under law, but civil liberties groups, journalists, and legal scholars continue to seek more information about what it entailed, who was targeted, and what authorities exist today for domestic intelligence-gathering.

Deep State has some answers.

Only about three dozen NSA officials have access to Ragtime’s intercept data on domestic counter-terrorism collection. That’s a tiny handful of the agency’s workforce, which has been pegged at about 30,000 people.

As many as 50 companies have provided data to this domestic collection program, the authors report.

If the NSA wants to collect information on a specific target, it needs one additional piece of evidence besides its own “link-analysis” protocols, a computerized analysis that assigns probability scores to each potential target. This is essentially a way to use a computer data-mining program to help determine whether someone is a national security threat. But the authors find that this isn’t sufficient if NSA wants to collect on said target. And while the authors found that the Foreign Intelligence Surveillance Court rarely rejects Ragtime-P requests, it often asks the NSA to provide more information before approving them.

How the surveillance is approved tells us a lot about the breadth of the NSA’s intelligence gathering. The court and the Attorney General both certify a slate of approved targets under Ragtime-P, the authors find. That includes a certain amount of “bulk data”—such as phone call logs and records—that can be collected around those targets. An NSA official told the authors that Ragtime-P can process as many as 50 different data sets at one time.

What happens next looks like a 21st-century data assembly line. At the NSA’s headquarters in Fort Meade, Maryland, a program called Xkeyscore processes all intercepted electronic signals before sending them to different “production lines” that deal with specific issues. Here, we find another array of code names.

Pinwale is the main NSA database for recorded signals intercepts, the authors report. Within it, there are various keyword compartments, which the NSA calls “selectors.”

Metadata (things like the “To” and “From” field on an e-mail) is stored in a database called Marina. It generally stays there for five years.

In a database called Maui there is “finished reporting,” the transcripts and analysis of calls. (Metadata never goes here, the authors found.)

As all this is happening, there are dozens of other NSA signals activity lines, called SIGADS, processing data. There’s Anchory, an all-source database for communications intelligence; Homebase, which lets NSA analysts coordinate their searches based on priorities set by the Director of National Intelligence; Airgap, which deals with missions that are a priority for the Department of Defense; Wrangler, an electronic intelligence line; Tinman, which handles air warning and surveillance; and more.

Lest you get confused by this swirl of code names and acronyms, keep this image in mind of the NSA as a data-analysis factory. Based on my own reporting, the agency is collecting so much information every day that without a regimented, factory-like system, analysts would never have the chance to look at it all. Indeed, they don’t analyze much of it. Computers handle a chunk, but a lot of information remains stored for future analysis.

So who is monitoring this vast production to ensure that the communications of innocent Americans aren’t spied on? Ambinder and Grady report that for the NSA’s terrorism-related programs, the agency’s general counsel’s office regularly reveals “target folders,” which contain the identities of those individuals who are under surveillance, “to make sure the program complied with the instruction to surveil those reasonably assumed to have connections to al-Qaeda.”

That the NSA is policing itself may come as small comfort to many critics of the Obama administration’s intelligence programs. The size of the “compliance staff” that monitors this activity is only about four or five people, depending on what’s available in the budget at any moment, the authors report. They also say that we cannot know whether the program is pushing beyond the boundaries of the law.

However, outside the closed circle of about three dozen NSA employees who are read in to Ragtime, there more than 1,000 people “outside the NSA are privy to the full details of the program.” If NSA is breaking the law, “how much longer can that secret last?” the authors ask.

We have a preceding example to test this hypothesis, albeit in a limited fashion. In 2004, the senior leadership of the Justice Department and the FBI threatened to resign over what they saw as illegal collection activities at the NSA, collection activities that are still going on under Ragtime and under new surveillance law.

Back then, James Comey, acting as Attorney General while John Ashcroft was in the hospital, refused to sign a set of certifications provided by the Justice Department to Internet, financial, and data companies, the authors report. Why? Comey believed that the justification for providing bulk data to the NSA wasn’t sufficient.

The administration’s tortured logic “drove him bonkers. There was just no way to justify this,” the authors report, quoting people who have spoken to Comey, who has never publicly said why he objected. Interestingly, the authors find that the parts of the program he was objecting to didn’t implicate the Foreign Intelligence Surveillance Act.

This comports with my own reporting in my book, The Watchers. The NSA was making “mirrors” of telecommunications databases, so that analysts could go through the data and mine it for clues. As it has been explained to me, the problem here dealt with how the government viewed its legal authorities to access data stored in computers, and whether analysts could dip back into it without specific authorizations. Importantly, this data consisted of that so-called “bulk data.” It wasn’t recorded phone calls or the text of e-mails. That information was governed by FISA–or should have been–because it was considered “content” under law, and that requires a warrant to obtain.

The White House panicked when Comey and Ashcroft refused to sign off, Ambinder and Grady report, fearing that the companies on which NSA was depending for information would cut the agency off if they didn’t get a signed order from the Attorney General himself. It took six months for the administration to reshape the program so that it comported with “interpretation of the metatdata provisions” that were promulgated by the Justice Department’s Office of Legal Counsel.

Had these officials resigned, it’s unthinkable that the secrets of NSA’s intelligence gathering activities would have stayed hidden. A year later, in 2005, they were revealed in part by the New York Times. Here, too, Ambinder and Grady have some new insights. It turns out that while the NSA’s director, General Michael Hayden, was publicly excoriating the newspaper for disclosing the classified activities, he was privately glad that they withheld what he considered key operational details.