Stunned and dejected by the death of a bill to restrain National Security Agency surveillance, civil libertarian groups vowed to return to the daunting effort in the next Congress.

The USA Freedom Act, a consensus bill that would have divested the NSA of the bulk US phone records database disclosed last year by the Guardian thanks to Edward Snowden’s leaks, failed by two votes to clear a procedural hurdle in the Senate late on Tuesday. Overwhelmingly, the bill’s opponents cited the prospect of a domestic terrorist attack from the Islamic State (Isis) as a motivator for their vote, even though not even the NSA continues to argue that the phone-data dragnet would prevent one.

Senator Ron Wyden, an Oregon Democrat and an ardent supporter of the bill, said privacy advocates “will not rest until there is true reform that ends this unnecessary overreach”.

The ACLU’s Washington director, Laura Murphy, agreed: “Though this vote is a setback, it will not stop the push for reform.”

The challenges that such reform faces are significant.

The Congress that ultimately rejected the USA Freedom Act did so in the aftermath of dramatic disclosures courtesy of the Snowden archive. Headlines about bulk surveillance have now been replaced by those about a war with Isis that is expected to last years.

The debate has eerie echoes of 2007, when, as detailed in a new book by Daily Beast reporter Shane Harris, intelligence officials warned Congress that failing to give NSA authorities the cover of law would prevent battlefield intelligence collection benefiting US troops in Iraq.

Yet a significant piece of leverage remains available to civil libertarians. In June, the provision of the Patriot Act cited by the government to justify the bulk domestic phone records collection, known as Section 215, will expire. Freedom Act advocates in the House of Representatives told the Guardian in October that they doubted the next House, which will be significantly more Republican, will cobble together the votes to pass a reauthorization.

Still, expiration of Section 215 would not address other aspects of the bill, such as adding an adversarial process to the secret Fisa court or lifting gag rules on tech companies and telecoms that receive surveillance orders. Additionally, the NSA has an established history of discovering new legal authorities for desired surveillance when losing others.

Senator Dianne Feinstein, a California Democrat and longtime Freedom Act skeptic, brought herself to support the bill during the cloture motion for precisely that reason. In a statement after the vote, Feinstein said she looked forward to “continuing to work on this issue”, though she will do so without the intelligence committee chair’s gavel.

In the House, the new chairman of the intelligence committee will be Devin Nunes, a California Republican, House speaker John Boehner announced on Tuesday. Nunes made waves earlier this year by calling Congressman Justin Amash, one of bulk surveillance’s most fervent GOP critics, “al-Qaida’s best friend in Congress”.

As of late Tuesday, the White House and the intelligence agencies, all belated supporters of the USA Freedom Act, did not respond to questions about whether they will seek legislation in the next Congress to divest the NSA of its domestic phone records database. President Barack Obama agreed in January to replace it, while seeking reauthorization every 90 days from the Fisa court for the bulk phone-metadata collection. The current authorization expires in less than a month.

Reauthorizing Section 215 will immediately top the priorities list for FBI director James Comey. The FBI relies on the provision to compel the production of a variety of business records for its counterterrorism investigations. Even before the USA Freedom Act failed to overcome the Senate filibuster, Comey publicly lamented the “pendulum” of a “post-Snowden” debate favoring privacy.

A clue as to the intentions of the newly appointed NSA director, Admiral Michael Rogers, may come this week. Rogers is scheduled on Thursday to testify before the House intelligence panel about cyber threats. Expanding data-sharing with private companies about network intrusions is the NSA’s major legislative priority, but the agency has lamented that passage has not been possible without first addressing the domestic phone records collection.

Representative Adam Schiff, a California Democrat on the committee, said late on Tuesday: “I expect that early next year we will again work to find a compromise that protects privacy while ensuring we have the tools to discover and defeat any plots against our nation.”

“The challenge of NSA reform didn’t end tonight. Congress has only months left to reauthorize Section 215 or let it, and its programs, expire. And there are several other authorities waiting for reform in the wings. The need for reform isn’t going away, it’s only growing,” vowed Amie Stepanovich, a lawyer with the digital-rights group Access.