To pull off the prisoner swap of five Taliban leaders for Sgt. Bowe Bergdahl, the White House overrode an existing interagency process charged with debating the transfer of Guantanamo Bay prisoners and dismissed long-standing Pentagon and intelligence community concerns based on Top Secret intelligence about the dangers of releasing the five men, sources familiar with the debate tell TIME.

National Security Council officials at the White House decline to describe the work of the ad hoc process they established to trade the prisoners, or to detail the measures they have taken to limit the threat the Taliban officials may pose. They say consensus on the plan was reached by the top officials of Obama’s national security team, including representatives from the Pentagon, State Department, intelligence community and Joint Chiefs of Staff. “These releases were worked extensively through deputies and principals,” says National Security Counsel Deputy for Strategic Communications Ben Rhodes. “There was not a dissent on moving forward with this plan.”

But officials in the Pentagon and intelligence communities had successfully fought off release of the five men in the past, officials tell TIME. “This was out of the norm,” says one official familiar with the debate over the dangers of releasing the five Taliban officials. “There was never the conversation.” Obama’s move was an ultimate victory for those at the White House and the State Department who had previously argued the military should “suck it up and salute,” says the official familiar with the debate.

Obama has broad authority under Article II of the U.S. Constitution to order the prisoner exchange as commander in chief of America’s armed forces. The lengths to which he went to bring it about show how determined he was to resolve the lingering issue of America’s only prisoner of war in Afghanistan.

The Obama administration first considered whether the five men were safe to release at the very start of his term as president. In January 2009, Obama ordered a Justice Department-led review of all 240 Guantanamo Bay detainees. The five Taliban leaders were found to be high risks to return to the fight against Americans, confirming Bush administration assessments of the threat they posed, according to officials familiar with the group’s findings. “These five are clearly bad dudes,” says a second source familiar with the debate over their release, adding that the detainees are likely to return to the fight.

Thereafter, the administration established a regular process for handling the release of detainees from the military prison at Guantanamo Bay. Releases were considered and approved through the “Guantanamo Transfer Working Group” which comprised officials from the State Department, the Pentagon, the intelligence community, the Department of Homeland Security and the Joint Chiefs of Staff. Over time, 82 detainees have been released by the Obama administration, according to the latest report to Congress by the Office of the Director of National Intelligence.

The question of the release of the five Taliban leaders was a recurrent subject of debate in the administration and was a key element of the behind the scenes effort by the State Department and the White House to negotiate a peace deal with the Taliban. The transfer of the five was discussed as a possible confidence-building measure to pave the way for a deal. The debates over their release were contentious, officials familiar with them say.

Those opposing release had the benefit of secret and top secret intelligence showing that the five men were a continuing threat, officials familiar with the debate tell TIME. But in the push from the White House and the State Department to clear the men, opponents to release found themselves under constant pressure to prove that the five were dangerous. “It was a heavy burden to show they were bad,” says the second source familiar with the debate.

Opponents of release say absent a peace deal with the Taliban, the release makes no sense. “When our military is engaged in combat operations you’re always going to err on the side of caution,” says the first official familiar with the debate. “Just conceptually, how much sense does it make to release your enemy when you’re still at war with him?”

During previous debates, opponents were aided by a law passed by Congress during Obama’s first term that required the administration to certify to a set of onerous conditions that the administration said were nearly impossible to meet. That changed thanks to the efforts of Sen. Carl Levin, who managed to weaken certification standards in the 2014 National Defense Authorization Act to allow the Secretary of Defense to release Guantanamo prisoners when it is in the national security interests of the United States.

That change made it easier for the President to exert his commander in chief powers in effecting the prisoner swap. So far the White House has said little about the measures they negotiated to assure the men would not be a threat upon release. Administration officials have said the men will remain in Qatar under a one-year travel ban. Under existing procedure, released detainees are monitored by the CIA station chief in the country where they reside. On Tuesday, Obama said he had confidence the U.S. would “be in a position to go after them if in fact they are engaging in activities that threaten our defense.”

But Republicans now question whether the president has gone too far, even under the new law, which still requires 30 days’ notice ahead of a release from Guantanamo Bay. Administration officials told members of the Senate armed services and intelligence committees “repeatedly they weren’t going to [release the five men] and they would be notified and consulted if they did,” says a GOP Senate aide. The committees were only notified after the fact.

At least one member of the Senate did have advance notice. “We were notified of the plan to secure Sergeant Bergdahl’s release on Friday,” said Adam Jentleson, a spokesman for Senate Majority Leader Harry Reid. A spokesman for Republican House Speaker John Boehner, however, told TIME that there was no advance notice given to the leader of the House. Senate Intelligence Chair Dianne Feinstein was not informed in advance, either, and on Tuesday Deputy National Security Advisor Tony Blinken called her to apologize for the oversight, she told reporters.

The White House said Tuesday the President had exercised his constitutional authority out of a sense of urgency for Bergdahl’s safety. “Delaying the transfer in order to provide the 30-day notice would interfere with the Executive’s performance of two related functions that the Constitution assigns to the President: protecting the lives of Americans abroad and protecting U.S. soldiers,” National Security Council spokeswoman Caitlin Hayden said in a statement released to the press. “Because such interference would significantly alter the balance between Congress and the President, and could even raise constitutional concerns, we believe it is fair to conclude that Congress did not intend that the Administration would be barred from taking the action it did in these circumstances,” Hayden said.

Jack Goldsmith, a Bush administration veteran of the battles between the executive branch and Congress over Commander-in-Chief powers in the war against terrorists, says Obama may have been acting legally. On the website Lawfare Tuesday he wrote, “If the statute impinged on an exclusive presidential power, the president properly disregarded it and did not violate it.”

Even many of those who opposed the release in the past accept the president has the power in conflicts to effect a prisoner swap. “We have done prisoner swaps in the past,” says the first official familiar with the debate over the release. But, the official added, “That’s been in international armed conflict where you have a state with which you can negotiate and you can say this guy will not go back to the fight.”

–with additional reporting by Zeke Miller and Alex Rogers/Washington

Contact us at letters@time.com.