GUANTANAMO BAY NAVAL BASE, Cuba - Some of the men held here for more than a decade have been drafting plans for work and marriage on the outside or studying languages, preparing for a not-too-distant future beyond the coiled razor wire that surrounds the U.S. prison perched at the edge of the Caribbean Sea.

Until the past week, they had good reason to believe their ticket out might be imminent, if not home then at least to another country. President Barack Obama and others in the administration say they are committed to closing the Guantanamo detention center and military officials say they can resume transfers at a moment's notice, just as they did with the May 31 swap of five Guantanamo inmates for a captured American soldier.

"All I need is the names and a country and we could do it all very, very efficiently," the commander of U.S. Southern Command, Marine Gen. John F. Kelly, said in an interview Saturday at the start of a visit to the base he oversees.

But the current furor over the trade of the five Taliban prisoners for American Sgt. Bowe Bergdahl may have complicated the situation.

Get Breaking News Delivered to Your Inbox

The deal to swap Bergdahl, who was held by the Taliban for five years in Afghanistan, was brokered by the Obama White House without consulting Congress. Both Republican and Democratic lawmakers who initially praised Bergdahl's release have since backed off amid an outcry over the exchange, including questions about whether he walked away from his post before he was captured.

Congress plans hearings on the exchange for the five prisoners, who officials here say were leaders inside the detention center as well as in the Taliban. Any immediate transfers of other inmates are likely to further inflame members of Congress, much to the dismay of attorneys for some of the 71 prisoners awaiting transfer after a security review.

Before anyone can be released, the Obama administration must obtain security and humane treatment assurances with the home country or repatriation agreements for third countries, a time-consuming process even before the required 30-day notice to Congress, which eased the restrictions on transfers last year but still bars sending any of the men to the United States.

"It's unfortunate that cleared people may well suffer because of the backlash over this," said Cori Crider, an attorney for the British legal rights group Reprieve who represents several men approved for transfer. "I hope the more sensible people in Congress - as well as the White House and the Defense Department who recognize that it's time to get this done, that it's a stain on America's image - aren't going to be bullied."

The U.S. currently holds 149 men at Guantanamo. They include five prisoners charged with planning and aiding the Sept. 11, 2001, terrorist attack who face trial by military commission, as well as a handful of others being prosecuted. Most have been held without charge since the government began taking prisoners suspected of links to al Qaeda and the Taliban at the Navy base on the southeastern edge of Cuba in January 2002.

New York Times columnist Thomas Friedman told CBS' "Face the Nation" on Sunday that too much has been made of the prisoners at Guantanamo Bay.

"We have have built these people up into giants," Friedman said. "I'm going to sleep well tonight whether these guys whether these guys are in Qatar, whether they're in the Ritz there, or whether they're around the block."

Appearing on the same show, conservative pundit Peggy Noonan said her main concern was that the Obama administration did not appear to have a plan for what to do with the men who were released in terms of assuring they would not become problems down the line.

To Kelly, men held here are "some of the most despicable and dangerous men on the planet," as he put it in an impassioned speech Saturday night praising the troops who guard prisoners during a ball held at the base to mark the founding of the U.S. Army in June 1775.

The general declined in an interview with The Associated Press to discuss the exchange for Bergdahl. He did say the five Taliban were influential at Guantanamo. They were held in the communal unit known as Camp 6, where detainees can while away their time with up to 18 hours of recreation per day, including more than 100 satellite TV channels and other privileges as long as they follow the rules.

"These guys were leaders but they were smart leaders," Kelly said. "They liked the easy lifestyle of the communal (unit) and encouraging other members of the detention population to act out."

Attorney Ramzi Kassem, who has represented prisoners at Guantanamo for nearly a decade, said he was told by one client that just before the five Taliban were transferred the prison was locked down in unusually tight security for what they were told was the approach of a hurricane.

"The prisoners saw right through that and knew something big was up," said Kassem, a law professor at the City University of New York. "Within a day or two of the event, everyone knew."

The mood appeared calm on Saturday, the first time a journalist was allowed inside the detention center since the swap. Prisoners were seen in Camp 6, where nearly 80 men are held, listening to headphones and chatting in a communal pod. There were no obvious signs of strife in Camp 5, which is used for prisoners who are considered "non-compliant" and holds about 60-70 men in a stricter, single-cell environment. Journalists are not permitted to see Camp 7, home to the Sept. 11 defendants and others deemed "high-value" by the Pentagon.

The release of the five Taliban has created a sense of optimism, at least for now, said the prison's Muslim cultural affairs adviser, who can only be identified by the nickname "Zak," under military security rules. "They are looking at it as what could be the opening of the door for future transfers," he said. "They are just analyzing it like everybody else."

Kassem said he wants to see the Obama administration "bear down on the Defense Department" and resume transfers. "It is sickening that some politicians are trying to scapegoat Guantanamo prisoners who have been cleared for release," he said.