Incoming FBI chief threatened to quit over illegal surveillance but NSA continued to harvest data after his act of rebellion

James Comey famously threatened to resign from the Justice Department in 2004 over the warrantless surveillance of Americans' internet records. But once Justice Department and National Security Agency lawyers found a novel legal theory to cover the surveillance, the man Barack Obama tapped last week to lead the FBI stayed on as deputy attorney general for another year as the monitoring continued.

Comey was the acting attorney general in March 2004, when long-simmering legal tensions over the online "metadata" surveillance pitted the Justice Department and FBI against the Bush White House and NSA. That incident, dramatically recounted by Comey to the Senate in May 2007, earned the 6ft 8in former federal prosecutor a reputation for integrity that has become central to his persona.

President Obama directly referred to that reputation when he nominated Comey to take over the FBI on June 21. Hovering over the announcement were the Guardian and Washington Post's revelations of wide-ranging surveillance efforts.

"To know Jim Comey is also to know his fierce independence and his deep integrity," Obama said. "He was prepared to give up a job he loved rather than be part of something he felt was fundamentally wrong."

Except that a classified report recounting the incident, acquired by the Guardian, complicates that view. Comey threatened to resign over the perceived illegality of one aspect of the surveillance. But he remained at the Justice Department for another year as that effort, operating under a new legal theory, continued nearly unchanged.

The NSA surveillance program that began after 9/11, codenamed Stellar Wind, initially relied on an authorization signed by George W Bush in October 2001, largely penned by a legal adviser to Dick Cheney, the vice-president.

According to a draft NSA inspector general's report obtained by the Guardian, as well as a declassified history written by multiple US inspectors general, by late 2003, several senior Justice officials began to grow uncomfortable with the legal underpinnings of one of the four aspects of Stellar Wind – what the draft NSA report calls "bulk collection of internet metadata." Among them were Comey, assistant attorney general Jack Goldsmith, and FBI director Robert Mueller.

When attorney general John Ashcroft grew too sick to provide a periodic re-certification of the program's legality, White House and NSA officials turned to Comey, the acting attorney general, to bless it. Comey, backed by Goldsmith and Mueller, refused.

Secretly, a crisis ensued in March 2004. The three Justice and FBI officials threatened to resign if the program continued. White House aides Alberto Gonzales and Andrew Card visited Ashcroft as he recovered from surgery in his George Washington University hospital bed to see if Ashcroft would reverse Comey's decision. Ashcroft refused. To forestall Comey, Goldsmith and Mueller's resignations, Bush agreed to "discontinue" the NSA's bulk collection of Americans' online records. On March 26 2004, it stopped.

Comey would later testify to the Senate that the episode was "the most difficult of my professional career."

But "immediately," the NSA IG report shows, lawyers from the NSA and Comey's Justice Department "began efforts to recreate this authority." They found it in what the document nebulously refers to as a Pen Register/Trap and Trace Order – a reference to devices traditionally used by surveillance officials to record the incoming and outgoing calls made and received by a telephone.

The Fisa court, the secret court that oversees NSA surveillance, approved the first such order for NSA to again collect and analyze large volumes of internet records from Americans on July 14 2004, barely three months after Comey's rebellion.

"Although NSA lost access to the bulk metadata from 26 March 2004 until the order was signed, the order essentially gave NSA the same authority to collect bulk internet metadata that it had" previously, the NSA IG report reads, "except that it specified the datalinks from which NSA could collect, and it limited the number of people that could access the data."

The surveillance Comey and his colleagues – including Mueller, the FBI director he is nominated to replace – objected to had merely been paused and rerouted under a new legal basis. Comey remained at the Justice Department as deputy attorney general until August 15, 2005.

"Good lawyering is always better than bad lawyering, but it's really the conduct that shocks the American public," said Mike German, a former FBI agent and current lawyer for the American Civil Liberties Union.

"The troubling aspect of it is that the offensive conduct continued, albeit under a different legal regime," German said. "So it raises questions as to whether [Comey's resistance] was a matter of legal posturing to justify a program that on its face was constitutionally overbroad, or an effort to actually stop the collection. If it's the first, where the person is going into a position where he will be overseeing similar programs for the next 10 years, then that is more problematic than if it was to stop the conduct."