UNDATED - In this undated image provided by the U.S. Army, Sgt. Bowe Bergdahl poses in front of an American flag. U.S. officials say Bergdahl, the only American soldier held prisoner in Afghanistan, was exchanged for five Taliban commanders being held at Guantanamo Bay, Cuba, according to published reports. Bergdahl is in stable condition at a Berlin hospital, according to the reports. (Photo by U.S. Army via Getty Images) Army Sgt. Bowe Bergdahl poses in front of an American flag. U.S. officials say Bergdahl, the only American soldier held prisoner in Afghanistan, was exchanged for five Taliban commanders being held at Guantanamo Bay. (credit: U.S. Army via Getty Images)

WASHINGTON (CBSDC/AP) — White House aides are accusing soldiers who served with Army Sgt. Bowe Bergdahl are “swift boating” him following his release over the weekend after five years in captivity.

NBC News’ Chuck Todd reported Monday that the White House did not expect this sort of vitriolic backlash exchanging five high-level Taliban members held at Guantanamo Bay for Bergdahl, who left his Army post in 2009 in Afghanistan and was subsequently taken captive.

“They did not expect this backlash on Bergdahl himself,” Todd reported on NBC’s “Today.” I’ve had a few aides describe it to me as we didn’t know that they were going to ‘swift boat’ Bergdahl. And that’s a reference to that political fight back in 2004 over John Kerry’s military service.”

Bergdahl disappeared on June 30, 2009. A Pentagon investigation concluded in 2010 that the evidence was “incontrovertible” that he walked away from his unit, said a former Pentagon official who has read it.

The military investigation was broader than a criminal inquiry, this official said, and it didn’t formally accuse Bergdahl of desertion. In interviews as part of the probe, members of his unit portrayed him as a naive, “delusional” person who thought he could help the Afghan people by leaving his Army post, said the official, who was present for the interviews.

Reports have surfaced that at least six U.S. soldiers were killed while searching for Bergdahl.

Matt Vierkant, 27, of Sioux Falls, South Dakota, was a team leader of another squad in Bergdahl’s platoon and said soldiers from his unit and other units were wounded or killed on missions to chase down leads related to Bergdahl.

Asked about the statement Sunday by National Security Adviser Susan Rice that Bergdahl served “with honor and distinction,” he said: “That statement couldn’t be further from the truth. I don’t know if she was misinformed or doesn’t know about the investigations and everything else, or what.”

He said Bergdahl’s fellow soldiers knew within five or 10 minutes from the discovery of disappearance that he had walked away. In retrospect the signs were there, he said, but there was nothing so definitive that would have prompted action.

“He said some strange things, like, ‘I could get lost in those mountains,’ which, at the time, that doesn’t really strike you as someone who is going to leave their weapon and walk out.”

Vierkant said he believes it’s paramount that an investigation determine whether Bergdahl deserted or collaborated with the enemy.

“It shouldn’t even be a question of whether, it should question of when,” he said.

Republicans in Congress criticized the agreement and complained about not having been consulted, citing a law that requires Congress to be given 30 days’ notice before a prisoner is released from Guantanamo.

Republicans on the House Armed Services Committee said the Pentagon notified the panel by phone on Saturday that the exchange was occurring in the next five hours.

“A phone call does not meet the legal standard of congressional notification,” the Republican members said in a statement and added that official notice of the move came Monday, “more than 72 hours after the detainees were released.”

Republicans also argued that the swap could set a dangerous precedent.

“The five terrorists released were the hardest of the hard-core,” said Sen. Lindsey Graham, a South Carolina Republican. “I fear President Obama’s decision will inevitably lead to more Americans being kidnapped and held hostage throughout the world.”

Larry Sabato, director of the Center for Politics at the University of Virginia, told CBSDC that the administration tried to sell this “great story” of praising Bergdahl as a hero.

“It didn’t take long for this PR push to falter. His fellow soldiers describe Bergdahl as a deserter, and they are bitter that American lives were lost trying to find and rescue him,” Sabato explained to CBSDC. “And then there’s the prisoner swap. Those five senior terrorists caused a lot of people’s stomachs to turn and temperatures to rise. Quite a price to pay for a soldier who may be a deserter, if reports are true.”

During a news conference in Poland Tuesday, Obama defended his decision of swapping the Taliban prisoners for Bergdahl, saying they were worried about his health after spending the past five years in captivity.

“Regardless of the circumstances, whatever those circumstances may turn out to be, we still get an American soldier back if he’s held in captivity,” Obama said. “We don’t condition that.”

Despite what some lawmakers are saying that they were being kept in the dark, Obama stated that the White House had been consulting with Congress “for some time” about the proposed deal.

“We seized that opportunity,” Obama said.

Obama also noted that it is entirely possible that the released detainees, who are mandated to stay in Qatar under the agreement for the next year, could return to terrorism.

“We will be keeping eyes on them,” Obama said. “We have confidence that we will be in a position to go after them if in fact they are engaging in activities that threaten our defenses.”

(TM and © Copyright 2014 CBS Radio Inc. and its relevant subsidiaries. CBS RADIO and EYE Logo TM and Copyright 2014 CBS Broadcasting Inc. Used under license. All Rights Reserved. This material may not be published, broadcast, rewritten, or redistributed. The Associated Press contributed to this report.)