Washington (CNN) The release of three Americans held in North Korea gives the Trump administration a feel-good foreign policy victory -- but it also highlights the bleak prospects for Americans held in Iran now that the US has left the Iran nuclear deal.

While announcing that he was pulling the US out of the deal, which curbed Tehran's nuclear program in exchange for sanctions relief, Trump announced that Secretary of State Mike Pompeo was in North Korea and teased the release of the three detainees.

Trump excoriated Iran for its "destabilizing activities" in the region in his remarks from the White House Diplomatic Room, including Tehran's "support for terrorism," its "nuclear threat" as well as the dangers posed by its ballistic missile program. And while he warned that, "America will not be held hostage to nuclear blackmail," he did not mention the Americans being held in Iran.

"These people are being lost in the shuffle," Jason Rezaian, a Washington Post reporter imprisoned in Iran for about 18 months, told CNN's Wolf Blitzer after Trump announced his decision. Rezaian added that he didn't see "any way out now" for those still behind bars in Iran.

Lost in the shuffle

US officials estimate there are about six Americans being held in Iran, though experts on the country say the number is probably higher, as some families choose not to publicize their loved ones' situation in the hope that a quiet approach will make for a quicker release.

The numbers include four Iranian-Americans and one Chinese-American graduate student accused of various crimes, including "collaborating with enemy states" and spying. They include businessman Morad Tahbaz, gallery owner Karan Vafadari, Baquer and Siamak Namazi , a father and son, and Xiyue Wang, a Princeton PhD candidate.

Perhaps the highest profile American held by Iran is Robert Alan Levinson, a retired FBI agent working for the CIA, who disappeared while on Kish Island in Iran in 2007.

Iran's Foreign Minister Mohammad Javad Zarif told CNN in a 2014 interview that he had not seen evidence that proves Levinson was even in Iran. In April, Zarif suggested during an interview with CBS' Face the Nation that Iran would be open to a prisoner swap if Trump showed Iran more respect and stopped making references to regime change.

A swap "is a possibility, certainly from a humanitarian perspective, but, but it requires a change of attitude," Zarif said, "and a change of language."

There's no evidence that Trump will offer that kind of change any time soon, and former officials who have dealt with Iran were pessimistic about the prospects for Americans held in the country.

Trump's decision to leave the deal "increases the risks for those loved ones and it increases the likelihood that we will not see them come home anytime soon," said Wendy Sherman, a former Under Secretary of State for Political Affairs who helped negotiate the Joint Comprehensive Plan of Action, as the Iran deal is formally known.

"It is absolutely incumbent upon our government to do everything we can to make sure Americans are not left behind, and that we bring the Americans who are detained there home, that we find Robert Levinson and bring him home," Sherman said during a call with reporters arranged by the group National Security Action.

She noted that it has been "years that the Levinson family has been struggling with not knowing where Bob is, and I think it will be particularly painful to see the detained in North Korea come home without North Korea really having done anything, where Iran has fully complied with the Joint Comprehensive Plan of Action since January of 2016, nearly 2 years of full compliance. And yet Americans are still stuck."

JUST WATCHED The mysterious disappearance of Bob Levinson Replay More Videos ... MUST WATCH The mysterious disappearance of Bob Levinson 02:43

Moderates in Iran who might have wanted to help Americans get home now have little political power to make that happen, Sherman noted, as now it would look like they were making concessions to the US without getting anything in return, just as Trump is vilifying the country.

Trita Parsi, president of the National Iranian American Council, noted that previous detainee releases from Iran came about because of efforts by another country or direct talks between Iran and the US. He noted that the only regular channel of communication between Tehran and Washington - quarterly talks associated with the JCPOA - will no longer take place now that the US has pulled out of the deal.

"The only regular channel that existed between the US and Iran was the JCPOA," Parsi said. "Now that the US has violated that and pulled out of it, it's difficult to see how we're going to get those Americans out."

"We're doing our best"

As Pompeo flew from North Korea to Japan with the three Korean-American detainees on Wednesday, he was asked about the perception some might have that the US was leaving some Americans behind while picking others.

"We're doing our best to get them all back," Pompeo said. "There are Americans being held in several places, right. There are Americans detained in Syria as well. When I was the CIA director, I could see the State Department and all of the United States Government was focused on getting every one of those Americans returned wherever they were."

Reporters in Washington asked a senior State Department official a similar question Tuesday, pushing for an answer on what the administration will do to help those detained in Iran.

"Let me offer that the security and safety of Americans is our top priority," the official said. The official had "just come" from a White House call with the family of one of the Americans detained in Iran. "They were asking exactly the same things," the official said.

"I don't want to - this is going to be unsatisfying to you - I don't want to get into it," the official said. "That's conversations we're having with a number of parties to try and resolve those cases, but - so I'm leery. I'm leery about getting too much into that, and I know that's unsatisfying, so I'm sorry."