President Obama released five Taliban leaders from the Guantanamo Bay detention facility on Saturday without consulting Congress and without strict assurances that the militants won’t somehow return to the fight. Republicans on Capitol Hill worry that the swap of these Taliban leaders for American hostage Sgt. Bowe Bergdahl is a prelude to a bigger move—the emptying out of Guantanamo entirely.

In his 2014 State of the Union address, Obama promised to shutter the prison built on Cuban soil by the end of the year. Obama now has seven months to fulfill his latest promise to shut down Guantanamo—or come as close to it as he can. During that time, Congress will be unable to prevent the release of the 149 prisoners still there.

“This whole deal may have been a test to see how far the administration can actually push it, and if Congress doesn’t fight back they will feel more empowered to move forward with additional transfers,” said one senior GOP senate aide close to the issue. “They’ve lined up all the dominoes to be able to move a lot more detainees out of Guantanamo and this could be just the beginning.”

On Saturday, only three days after announcing the complete withdrawal of U.S. troops from Afghanistan by 2017, the White House revealed that it was releasing the Taliban commanders in exchange for Bergdahl. The law requires 30 days’ advance Congressional notification before such a release from Guantanamo. But it was simply not workable in this case, White House officials, said; the soldier’s health was failing, and the Qatari-brokered deal for the prisoner swap unfolded too quickly.

Lawmakers’ surprise fueled their confusion and skepticism about the deal—especially since the on-again off-again negotiations for Bergdahl were an open secret, dating back years. GOP leaders, in particular, were offended that they received calls from their Democratic counterparts and administration officials only after Bergdahl’s release was reported in the news.

Now there’s growing concern on Capitol Hill that President Obama intends to bypass Congress to fulfill his promise to close the prison by releasing scores of more Guantanamo prisoners with little public or even private debate. Lawmakers and staffers see the Bergdahl case as only the latest maneuver in a larger plan to cut Congress out of the Guantanamo issue; and they’re not exactly reassured by senior administration officials’ refusal to disclose what steps will be taken to mitigate the risk that these prisoners could become involved again in the Afghan insurgency.

Of course, the Bergdahl case is a special one. Partially due to concern over Bergdahl’s fate, Congress actually gave up a significant amount of oversight of Guantanamo releases last December by passing a defense policy bill that eased the burden on the administration before releasing prisoners. Now, it’s just a simple Congressional notification. The law itself contains no enforcement measures and Obama even issued a signing statement at the time saying even the remaining restrictions violated his Constitutional prerogative.

(Candidate Obama pledged to do away with signing statements as an unconstitutional “end-run around Congress,” but often in these matters, where you sit is where you stand.)

“The whole reason the administration ignores reporting requirements is because they know there is no consequence for ignoring them,” the senior GOP senate aide said.

It’s also true that the Obama administration has been steadily transferring Guantanamo prisoners to third countries for years, as the George W. Bush administration did before it. But the Bergdahl deal was the first major release under the new, looser restrictions and the five Taliban former prisoners were seen as some of the worst of the worst being held at the facility.

Despite outrage by several Republican leaders—including House Armed Services Committee Chairman Buck McKeon and Senate Armed Services Committee Ranking Republican James Inhofe —about the release of the Taliban commanders, privately GOP offices admit that the deal is done. All they can do now is demand more transparency. But the GOP does intend to ramp up the fight over the larger issue of releasing the other 149 prisoners by taking trying to take some of their oversight authority back.

“At this point, I don’t think Congress can do anything on his case. Nothing Congress is going to do is going to bring these guys back. But what this might lead to is tightening restrictions overall to make it harder for the administration to do this again,” another senior GOP senate aide said.

Behind the scenes, several GOP senators had already been working to further restrict the Obama administration’s authority to release Guantanamo prisoners, even before the news of the Bergdahl prisoner swap broke. In a previously unreported closed-door meeting of the Senate Armed Services Committee over next year’s defense policy bill, a fierce debate erupted over GOP efforts to slow the release of the prisoners.

Committee leaders Carl Levin and Inhofe had already included in their original version of the bill a provision requiring the administration to report on its overall strategy for handling the prisoners, the risk they posed if released, and their overall plan to close the prison before releasing the rest of the detainees. In the closed session, Sen. Lindsey Graham won support for an amendment that would provide for Congress to vote on that strategy on the Senate floor if it ever surfaced.

Then, Sen. Kelly Ayotte narrowly won support for her amendment, an outright ban on transferring Guantanamo prisoners to Yemen for one year. If implemented, that would effectively stop Obama from closing the prison in 2014. Over half of the remaining prisoners are Yemeni, and would presumably be shipped back there. Several Yemeni Guantanamo alumni released over the years have now reportedly taken up leadership positions in an al Qaeda affiliate there.

The GOP effort, however, will take longer than seven months to have any effect because the defense policy bill won’t even be considered in the Senate until after the election. Typically, the defense bill is passed in late December. There’s no chance Congress will pass an appropriations bill this year, meaning Congress can’t remove the funding for transfers. That gives Obama plenty of time to use the current looseness of the law to push forward the releases of many more prisoners.

The Republican-controlled House already passed its version of the defense policy bill and won’t get another chance to amend it until a potential conference with the Senate, after they pass their bill. House GOP aides said it’s unlikely the House would take up a stand-alone measure on Guantanamo prisoner releases and even if that passed, the Senate Democratic leadership would never take it up.

The drive to stop Obama from releasing Guantanamo prisoners without Congressional oversight over the next seven months has also lost its key Democratic ally, Senate Intelligence Committee Chairwoman Dianne Feinstein, who celebrated the Bergdahl-Taliban swap this weekend.

“I support the president’s decision, particularly in light of Sgt. Bergdahl’s declining health. It demonstrates that America leaves no soldier behind,” she said in a statement.

That’s a change of position for Feinstein. Back in 2012, when the prisoner swap negotiations were done directly by State Department, Pentagon, and White House officials in secret meetings with the Taliban in Berlin and Doha, Feinstein was vehemently opposed to the idea.

“These are major Taliban figures, they are not minor people. And they will not be in the same kind of custody, maximum-security custody. Forget that it won’t be Guantánamo, just maximum-security custody,” she said at the time. “And in my view, there’s no way of knowing what they may do and what kind of propaganda they may breed.”