27 November stands to mark a victory a decade in the making for Merrill, who was served with a national security letter in 2004 and said he was ‘terrified’ of revealing the extent of the information he was being asked to hand over

For over a decade, Nick Merrill could say nothing about the FBI subpoena he received in 2004 – not to his wife, his ailing father nor his closest friends. Merrill didn’t even know if consulting an attorney about it would land him in a jail cell.

But now, after more than a decade of court challenges, Merrill is on the verge of revealing an unprecedented amount of detail about what the FBI and its partner agencies believe they can obtain without a warrant, using a controversial form of nonjudicial subpoena called a national security letter (NSL).

Unless the US Justice Department challenges a federal judge’s order that was kept secret until Monday, 27 November will be a landmark date for transparency in the post-9/11 era. The black bars obscuring the specific kinds of data the FBI sought from Merrill in 2004 will lift, revealing categories of information the bureau seeks to acquire outside the normal warrant process.

Until then, Merrill is unable to describe precisely what data the NSL covered. Its target, he said, was a “person”, whom the FBI is no longer investigating. But for Merrill, the decade-long legal odyssey has been a quest not only for transparency and privacy rights but also “to get my free speech back”.

Merrill, president of the firm Calyx, which at the time provided web hosting services, chose to fight the NSL instead of complying. He said that the public would be horrified to see the sorts of things the FBI acquires without a warrant.

“Internet service providers, email providers and other online services often store huge amounts of intensely personal and revealing data. NSLs have allowed the FBI to run rampant, demanding the sensitive records of innocent people in complete secrecy, avoiding the constitution’s carefully designed system of checks and balances, without ever appearing before a federal judge,” Merrill, now 42, told the Guardian.

“The Justice Department has fought for more than a decade to keep the American public from finding out what kinds of customer records it was secretly demanding that Nick Merrill turn over without a warrant. That these public resources have been used to keep Americans in the dark about how the government interprets the law is incompatible to democracy,” said former FBI agent Michael German.

A Justice Department official said the department is reviewing the judge’s order – which granted the government a 90-day period before the NSL’s disclosure – to determine whether it will challenge the release or permit NSL information to become public.

National security letters are a decades-old tool available to the FBI for obtaining what is typically and generically referred to as “subscriber information”, including phone records, email info or credit card data.

Across the country, the proliferation of NSLs are an abstract issue about the balance between privacy and security, between counterterrorism and centuries-old restrictions on government authority at its most coercive. But for Merrill it was highly personal, creating visceral divides between himself and everyone else.

“It sort of forces you to become an island unto yourself. You have all these internalized secrets you can’t talk to anyone about,” he said – secrets that became especially painful and isolating when his father contracted terminal cancer.

“I was staying at the hospice with my dad, and I couldn’t tell him that this significant portion of my life had been spent on this constitutional law battle,” Merrill remembered. “Both my grandmothers died. I never told them about it.”

Facebook Twitter Pinterest Merrill said of receiving the NSL: ‘It sort of forces you to become an island unto yourself. You have all these internalized secrets you can’t talk to anyone about.’ Photograph: Thatcher Cook for PopTech

On a redacted version of Merrill’s NSL, 11 lines specifying “electronic communication transactional records” are fully blacked out. Only disjointed phrases like “subscriber name”, “account number”, “address”, “telephone number”, “billing” and “email address” are visible. Its cover letter instructs Merrill to provide “transaction/activity logs and all email header information” as well.

Merrill was also instructed that applicable law “prohibits any officer, employee or agent of yours from disclosing to any person that the FBI has sought or obtained access to information or records under these provisions”.

“I was terrified,” Merrill said. “The letter didn’t even say what the consequences were if I didn’t do what they said. George Bush had said he could declare any American an enemy combatant and drag them to Guantánamo.”

The target of a federal investigation is rarely, if ever, the recipient of an NSL. Instead, the FBI requests data from the banks, telecoms or internet providers with which the target does business. The subpoenas, originally intended for exceptional circumstances to thwart espionage, are not issued by a judge or grand jury, but by FBI agents. They come with a gag order attached, preventing a company from alerting its clients.

The 2001 Patriot Act lowered the standards for issuing NSLs substantially. No longer did the FBI need a particularized suspicion against an individual target, just “relevance” to a terrorism or espionage investigation. Special agents-in-charge at the FBI’s 56 field offices could now issue an NSL, not merely senior officials at its Washington headquarters. The gag orders, however, remained.

Accordingly, the Patriot Act transformed NSLs from a rarity into a routine tool. While secrecy has made hard numbers about NSLs hard to acquire, the FBI made 8,500 requests for data through NSLs in 2000. In 2014 an advisory group on surveillance appointed by Barack Obama reported that the FBI now issues 60 NSLs on average every day, which works out to 21,900 annually. Since a single NSL can request records from multiple people, the total number of people affected is likely to be vastly higher – and remains obscured in secrecy.

The explosive growth of the secret subpoenas has alarmed tech companies and people outside of typical civil libertarian circles. A series of late-2000s US Justice Department inspector general reports found that NSLs seeking Americans’ data grew from 39% of NSLs in 2003 to 57% of those issued in 2006, an elision of normal warrant requirements. On occasions, the inspector general found, the FBI was “in violation” of NSL restrictions, and would seek NSLs after the secret Fisa court, a typically deferential body, turned down requests.

Obama’s surveillance advisory group, which included former US intelligence officials, warned that NSLs sought for intelligence investigations “are especially likely to implicate highly sensitive and personal information and to have potentially severe consequences for the individuals under investigation”. It said it was “unable to identify a principled reason why NSLs should be issued by FBI officials”.

Merrill, however, kept chipping away at the edifice of NSL secrecy. Usually known as John Doe, he was part of an early post-Patriot Act ACLU suit challenging the limits of the gag order. And he began to win: he was able in 2014 to acknowledge himself as the recipient of the NSL, and in December of that year exercised an annual option to challenge residual secrecy.

As Merrill sought to disclose more, the FBI said, according to New York federal judge Victor Marrero, Merrill “would reveal law enforcement techniques that the FBI has not acknowledged in the context of NSLs, would indicate the types of information the FBI deems important for investigative purposes, and could lead to potential targets of investigations changing their behavior to evade law enforcement detection”.

Marrero, on 28 August, ruled in Merrill’s favor anyway. But the judge’s opinion was kept secret until Monday to ensure that it did not itself reveal classified information.

“The gag was never necessary to protect national security, but to prevent a fulsome debate about the scope of the government’s unchecked power to obtain personal information about Americans,” said German, the former FBI agent, now with the Brennan Center for Justice at NYU Law School.

“Its release should be just the first step in a much more comprehensive discussion about the full range of government programs to spy on its own citizens.”