The members of president Barack Obama’s surveillance review panel on Tuesday rejected some of the central contentions offered by the National Security Agency for its bulk collection of phone records, including the program’s potential usefulness in preventing the 9/11 attacks.

Testifying before the Senate judiciary committee, members of the panel said that restricting the NSA is necessary in order to rebalance the competing values of liberty and security.

Richard Clarke, who was the White House's counter-terrorism czar on 9/11, echoed the 9/11 Commission in saying that the biggest obstacle to preventing the terrorist attack was not the NSA collecting an insufficient amount of data, but a failure to share information already collected.

“If the information that the federal agencies had at the time had been shared among the agencies, then one of them, the FBI, could have gone to the Fisa Court and could have in a very timely manner gotten a warrant to monitor” US-based al-Qaida conspirators, Clarke told the Senate judiciary committee.

Similarly, Michael Morell, a former deputy CIA director, told the committee that so-called “metadata” about a phone conversation inherently entailed information about the substance of the communication. “There is quite a bit of content in metadata,” Morrell said. “There’s not a sharp distinction between metadata and content. It’s more of a continuum.”

Morrell added that the bulk collection of domestic phone data “has not played a significant role in preventing any terrorist attacks to this point,” further undercutting a major rationale offered by the NSA since the Guardian first revealed the bulk phone-data collection in June, thanks to leaks by Edward Snowden.

But, Morell added, “that is a different statement than saying the program has not been important.” Morrell said that bulk collection can provide a reassurance that there is no domestic nexus to foreign terrorist plots detected by other NSA efforts.

“It is absolutely true that 215 has not by itself disrupted prevented terrorist attacks in the United States, but that doesn’t mean it’s not important going forward, said Morell, using a shorthand for the bulk phone metadata collection. “Many of us have never suffered a fire in our homes but many of us have homeowners insurance.”

The recommendations that the panel made in December recast the Washington debate over the NSA's mass surveillance activities and gave reform efforts crucial political momentum. Obama will likely announce some curbs to surveillance, heavily influenced by the panel’s work, in a speech at the Justice Department scheduled for Friday.

Former CIA deputy director Michael Morell said 'There's not a sharp distinction between metadata and content.' Photograph: Mark Wilson/Getty Images

“We’re really having a debate about Americans’ fundamental relationship with their government,” said Patrick Leahy, the Vermont Democrat who chairs the Senate judiciary committee and is the co-author of a bill, the USA Freedom Act, to restrict NSA bulk surveillance.

The panel has also prompted fierce behind-the-scenes jockeying between the NSA and its critics surrounding the scope of its highest-profile recommendation: ending the NSA’s collection of data on every phone call made in the United States.

Several senators at the hearing expressed skepticism at the panel’s recommendations, including intelligence committee chairwoman Dianne Feinstein and South Carolina Republican Lindsey Graham, who both seemed to confuse the review group by pressing them on the bulk collection of metadata’s relationship to preventing terrorism.

Both sides are awaiting Obama’s thoughts on the subject, particularly concerning the legal standards and procedures by which NSA would be allowed to access records kept by phone companies for discovering terrorism connections, and how long companies or a private entity would be required to retain customer information. Those details will determine whether the mass surveillance actually ends or, as critics have warned, is simply outsourced.

Most members of Congress, whom the White House concedes will take the fore in codifying any new surveillance approach that Obama proposes, have not yet taken firm positions on what the scope of a privately-held phone records database should be. On Tuesday, Representative Adam Schiff, a member of the House intelligence committee, introduced a bill he said would “restructure” the phone data collection, without requiring companies to hold customer data longer than they currently do, and forcing the NSA to obtain a Fisa Court order for searching through the data in all but emergency cases.

Dianne Feinstein speaks with director of national intelligence James Clapper, NSA director general Keith Alexander. Photograph: Jason Reed/Reuters

“This idea gained new momentum last month, with the president’s NSA review panel’s endorsement that restructuring the program is both technically feasible and more protective of the privacy interests of the American people,” Schiff, a California Democrat, said in a statement.

The members of the surveillance review panel testified on Tuesday that they were not advocating that the government no longer examine the metadata, but only that it should have to obtain court orders based on specific suspicions of wrongdoing before they do so in non-emergency cases.

"There's no reason why getting a court order to query the metadata is any more impossible than getting a search warrant to search a home," Geoffrey Stone, a University of Chicago law professor, told the Senate panel on Tuesday.

Panel members praised what they described as robust safeguards surrounding the bulk metadata program, but added that they were insufficient to maintain public confidence in it after the revelation of its existence.

The NSA argues that it needs “the haystack” of all domestic phone records in order to spot connections to terrorism, as outgoing deputy NSA director John Inglis said Friday. But telecommunication and internet firms are balking at any requirement for holding customer data longer than the current 18 month average maximum, fearing increased legal and financial liability.

“Our members would oppose the imposition of data retention obligations that would require them to maintain customer data for longer than necessary,” a spokesman for CTIA-The Wireless Association, the cellular phone trade group, told the Associated Press Friday.

The phone companies “obviously rather would not hold the data”, Stone conceded.

“The concern of the fourth amendment, the concern of our constitutional history is that government can do far more harm if it abuses information in its hands than private entities can,” Stone told the panel.

But Senator Chuck Grassley of Iowa, the ranking Republican on the committee, warned that the private sector would have a difficult time securing the data, citing the recent massive breach of customer information from Target.

Clarke later replied that there was a “very significant information compromise at NSA,” and that he was unaware of “people’s phone records going into the public record when they were stolen from phone companies.” But major phone companies overseas, like Deutsche Telekom in Germany and Vodafone in Greece, have experienced major data break-ins.

Brough Turner, the chief technology officer of broadband company NetBlazr, noted that the private sector itself has similar concerns. “As an internet service provider, the best possible thing is to keep the absolute minimum data necessary to track the functioning of your infrastructure and service outages and complaints, because you lay yourself open for excess legal expenses. It’s a liability issue,” he told the Guardian.

Turner was part of a lobbying push that tech firms made on Capitol Hill on Monday and Tuesday in an attempt to convince legislators to back the USA Freedom Act. Participants said they warned senators and members of Congress against handing over a program functionally similar to the NSA’s to the phone companies.

“We delivered a very clear message that [that action] doesn’t solve the problem at all,” said Matthew Simons of software firm ThoughtWorks. “Certainly in the global sphere, to tell our customers in Brazil, ‘Don’t worry, it’s not the government anymore, AT&T has got your back’ … it doesn’t fly, it doesn’t fix the problem.”

Barack Obama is expected to state his stance on surveillance reforms this week. Photograph: Carolyn Kaster/AP

Simons said the tech coalition, which primarily targeted members of the Senate and House judiciary committees, which are considering the USA Freedom Act, argued that the broad reach of the NSA’s foreign surveillance was hurting US tech competitiveness overseas – a contention he said caused cognitive dissonance on Capitol Hill.

“A lot of the people that have been the biggest champions of business interests are also the same people who see a terrorist behind every bush,” he said. “Those people have a really strong values conflict, because when businesses come to you and say, ‘Stop surveilling entire countries, the entire population of the world, because it’s killing our business,’ I think they just kind of freeze up. People are really wrestling with this.”

Among those wrestling with surveillance reforms is John Bates, a federal judge and former presiding judge of the Fisa Court. In a rare moment of policy advocacy from a sitting federal judge, Bates sent a letter to Feinstein, released Tuesday, in which he rejected several reforms to the secret surveillance court proposed by Obama’s review group, citing the “burden” they would place on the judiciary.

Creating a permanent privacy advocate to argue before the Fisa Court is “unnecessary – and could prove counterproductive – in the vast majority of Fisa matters”, wrote Bates, who did say that the appointment of one in certain cases at the court’s discretion is “likely to be helpful”.

Bates also warned against placing a controversial FBI administrative subpoena known as a National Security Letter under the court’s purview, saying it would “fundamentally transform the nature of the [court] to the detriment of its current responsibilities.”

Expanding the declassification of the secret court’s rulings as a transparency measure, Bates wrote, “is likely to promote confusion and misunderstanding”.

Cass Sunstein, a former Obama White House adviser and surveillance advisory panel member, said he disagreed with Bates about judicial discretion for appointing a privacy advocate on a case-by-case basis on the grounds that it afforded judges too much power in cases with significant privacy interests at stake.

“We think that’s not consistent with our traditions,” Sunstein testified.