Two Americans swimmers in Rio were removed from their flight to the United States by Brazilian authorities Wednesday and had their passports temporarily confiscated amid an investigation into the alleged robbery of the swimmers during the Olympics, authorities said.

Gunnar Bentz and Jack Conger were awaiting takeoff when they were removed from their plane and briefly held at the airport, sources told NBC Sports.

Attorney Sergio Viegas, who is representing the pair, said they will not be allowed to leave Brazil until they testify about the robbery. Their passports were "temporarily confiscated by a court order," a detective with the Rio tourist police department said.

The swimmers, as well as teammate Jimmy Feigen plan to meet with officials on Thursday.

The developments came after a Brazilian judge issued an order to seize the passports belonging to American swimmers Ryan Lochte and Feigen, prohibiting the Olympians from leaving the country.

Lochte was already back in the United States, and his lawyer said Wednesday he was being treated like a "pawn" by Brazilian authorities to get out from under the "dark cloud around these Olympics." Feigen is still in Brazil, where Lochte's attorney told NBC he is stuck. Sources at the Rio airport told NBC that Feigen checked in online for his flight home but never arrived at the airport, where his passport would have been confiscated.

Lochte, Feigen, Bentz and Conger say they were robbed at gunpoint in a taxi Sunday morning as they returned to the Olympic Village from a party, several hours after the last Olympic swimming events were held.

Police investigating reports that the four swimmers were robbed at gunpoint in a taxi so far have found little evidence supporting the account.

Lochte told NBC's Matt Lauer Wednesday that he left Rio before hearing about the controversy and vehemently denied that he'd made up the story of a robbery -- even as he backed away from some details in his earlier account.

Lochte now says his taxi wasn't pulled over by men with a badge, but that they were robbed after stopping at a gas station. Lochte also said the assailant pointed a gun at him rather than putting it to his head.

JUDGE CITES INCONSISTENCIES

According to the warrant, issued by Judge Keyla Blac De Cnop, the pair gave conflicting accounts of the alleged armed robbery. In his deposition, Lochte told investigators the swimmers were held up by one armed robber who demanded he hand over all of his money. Lochte said he gave the alleged suspect $400.

But Feigen, in his deposition, told investigators that the group was help up by several assailants and that only one was armed, according to the warrant.

Lochte's lawyer Jeff Ostrow refuted the judge's allegations of conflicting stories between the quartet, telling NBC that he doesn't know what the other swimmers said, but "the story happened the way Ryan provided in his statement."

"That’s BS. Ryan has always maintained there were number of them. I don’t know if three or four, but a number of them. If one guy walked up to him, he probably would’ve kicked his ass," Ostrow said, noting the Brazilian police didn't speak English well.

The judge also cited inconsistencies with the athletes' demeanor upon arrival at the Olympic Village after attending a party hosted by Brazilian swimmer Thiago Pereira at the Casa França in Rio's upscale south zone.

"I noticed the alleged victims arrived with their physical and mental integrities unshaken, in fact, they were joking with one another," DeCnop said, citing surveillance video at the Olympic Village.

Surveillance video obtained by the Daily Mail Online Wednesday appears to show the teammates arriving at the village and forgetting to remove their belts as they go through security.

Ostrow told NBC Wednesday afternoon there was no question the robbery happened as Lochte described it.

Lochte is one of the most decorated Olympic swimmers in U.S. history. He's won 12 Olympic medals over four Games, including six gold. In Rio, he won gold as a member of the 4x200 meter relay team along with Bentz and Conger. Feigen won gold in the 4x100 meter relay.

Local police arrived at the athletes village Wednesday morning and asked to meet with Lochte and Feigen, and to collect their passports in order to secure further testimony, according to Patrick Sandusky, a spokesman for the United States Olympic Committee.

"The swim team moved out of the village after their competition ended, so we were not able to make the athletes available," Sandusky said. “Additionally, as part of our standard security protocol, we do not make athlete travel plans public and therefore cannot confirm the athletes’ current location."

He said the committee will continue to cooperate with Brazilian authorities.

Ostrow said Lochte had always planned on leaving Rio after his events, and was never told by Brazilian authorities to "stick around, we want to investigate more, don’t go anywhere."

LOCHTE EXPLAINS

The swimmers were reportedly unable to provide key details in police interviews. The judge's order did not mention Bentz or Conger.

Word of the robbery initially created confusion between Olympic and U.S. officials. An International Olympic Committee spokesman at first said reports of the robbery were "absolutely not true," then reversed himself, apologized and said he was relying on initial information from the USOC that was wrong.

Attempting to explain the confusion, Lochte told USA Today that he and his teammates, didn't initially tell the U.S. Olympic officials about the robbery "because we were afraid we'd get in trouble."

Lauer said Wednesday evening that Lochte told him that the swimmers were worried about breaking the swim team code until they realized they couldn't get in trouble because they were over 21.

Asked about the surveillance video showing the swimmers returning to the Olympic Village, Lochte said they were afraid and in shock, not joking, as some viewers allege. He explained that the athletes were able to get home in a taxi with some money Conger had kept from the robbers.

Lauer said that Lochte strongly denied that he was trying to cover up something else embarrassing that may have happened.

The incident got attention in part because of fears around street crime at the games. Athletes and visitors have been told to use caution around the Olympics. Street crime was a major concern of Olympic organizers, and Brazil deployed 85,000 soldiers and police to secure the games, twice as many as Britain used during the 2012 London Olympics.

A spokesman for the U.S. Consulate in Rio said Wednesday it was aware of the reports on the court order and encouraged "all parties to work with Brazilian law enforcement in this investigation."

"This is a complete circus, and shame on Brazilian authorities for trying to flip the script and make it sound like Ryan isn’t telling the truth," Ostrow said.

NBC's Eric Hinton contributed to this report.