“It would be no more appropriate to have our government turn their backs on their citizens than to turn their backs on those who serve,” Mr. Boyle’s father, Patrick, told The Associated Press.

Members of the Weinstein family have been outspoken in their anger about statements from President Obama; the national security adviser, Susan E. Rice; Defense Secretary Chuck Hagel; and other senior officials. The administration has argued that it quickly made the deal with the Taliban because Sergeant Bergdahl’s health was deteriorating. Officials also said they feared that their leverage in negotiating the release with the Taliban would be significantly reduced as the United States’ withdrawal from Afghanistan neared.

“My father will be 73 in July, has a serious heart condition and severe asthma and had been taking medication to manage those conditions, and we don’t know if he is getting his medications,” said Mr. Weinstein’s daughter, Alisa.

She said a video that the American government obtained in December showed that he was pale and gaunt, and his voice sounded weak. “It wasn’t difficult to see he is under immense physical strain, and clearly his health is failing,” she said. “If the window was closing for Sergeant Bergdahl, the window is closing for my father.”

Family members said Mr. Weinstein could have been included in a swap because he is believed to be in Waziristan, where Sergeant Bergdahl was held. When the Weinsteins met with State Department officials in January, they told the officials that if the United States were to make a deal for Sergeant Bergdahl, it should include all American hostages.

“I don’t want you to think the people we have dealt with haven’t been very good to us, but this is the way it went down,” Elaine Weinstein said. “When they said that Bergdahl was freed, I was thrilled for his family, but I also wondered why didn’t it include my husband.”

A senior administration official defended its handling of the Bergdahl deal and said not all the kidnappings could be handled at the same time. “This was a swap with the Taliban for an American serviceman they had in captivity during a war,” said the official, who asked not be quoted by name in talking about national security. “It’s separate from the very serious efforts we undertake to return other American civilians home to their families.”