ESPN senior writer Wayne Drehs reports on the investigation surrounding Ryan Lochte and the small chance he will face punishment if his claims are proven to be false. (1:42)

RIO DE JANEIRO -- A Brazilian judge ordered Wednesday that the passports of American swimmers Ryan Lochte and Jimmy Feigen be seized, confining them to the country as authorities investigate their claim they were robbed at gunpoint during the Rio de Janeiro Olympics.

Steve Lochte told The Associated Press by phone that his gold-medal-winning son called him Tuesday after arriving back in the United States and told him he was going to pick up his car and buy a new wallet to replace the one that he said was stolen during the robbery.

"I'm just happy he's safe," the elder Lochte said. "It was an unfortunate experience for him and the other three. I don't know what all the controversy is. They were basically taken out of the taxi and robbed. The main thing is he's very lucky that he's safe and that all they got was his cash and wallet."

The elder Lochte said his son's Olympic credentials and cell phone weren't taken during the incident early Sunday morning. He said he was sure Ryan had his passport or he would not have been allowed to board a plane.

Feigen's whereabouts could not immediately be confirmed, though he told the San Antonio Express-News he was still in Brazil.

The office of Judge Keyla Blank confirmed the order to seize their passports in a statement Wednesday.

The U.S. Olympic Committee said police went to the athletes village Wednesday morning to try to collect the passports, but the swim team had already moved out. Spokesman Patrick Sandusky declined to say whether Lochte and Feigen were still in the country.

"As part of our standard security protocol, we do not make athlete travel plans public and therefore cannot confirm the athletes' current location," Sandusky said. "We will continue to cooperate with Brazilian authorities."

The U.S. State Department issued a statement Wednesday encouraging those involved to cooperate with Brazilian law enforcement.

Lochte and three of his teammates say they were robbed at gunpoint in a taxi Sunday morning as they returned to the athletes village from a party, several hours after the last Olympic swimming events were held.

Police have found little evidence so far to support their accounts, and say the swimmers were unable to provide key details in police interviews.

"Why would anybody fabricate anything?" Steve Lochte said. "It's just ridiculous."

A police official with knowledge of the investigation told The Associated Press that police cannot find their taxi driver or witnesses to the robbery. The official spoke on condition of anonymity because the investigation is ongoing.

Ryan Lochte swam in two events at the Rio Games, winning gold in the 4x200-meter freestyle relay. AP Photo/Michael Sohn

Lochte's attorney, Jeff Ostrow, has said there is no question the robbery happened.

"My opinion is they are trying to use Ryan's situation to use as a scapegoat to show that things down there are not as bad as it looks," Ostrow told ESPN on Wednesday. "These types of things have happened with other people, like the New Zealand athlete, but they don't take half the interest they do with Ryan.

"If they don't like the level of the detail that they've gotten from the statement ... then call me and we can give them more. Ryan is the victim here. You take one of the Brazilian authorities and put a gun to their head on a random street in America in the middle of the night and I guarantee they wouldn't be absolutely accurate with all the details.

"They can do and say whatever they want down there. The fact remains they were robbed at gunpoint."

According to statements from Lochte and the USOC, the swimmers were returning to the athletes village by taxi after a night out at the French Olympic team's hospitality house in the Rodrigo de Freitas area in the city's upscale south zone.

Traveling with Lochte and 26-year-old Feigen were 21-year-old Gunnar Bentz and 20-year-old Jack Conger. Lochte swam in two events at the Rio Games, winning gold in the 4x200-meter freestyle relay. He is a 12-time Olympic medalist.

Bentz's father, Steve Bentz, declined comment when reached by the AP.

The group did not call police, authorities said, and officers began investigating once they saw media reports in which Lochte's mother spoke about the robbery. Police interviewed Lochte and one other swimmer, who said they had been intoxicated and could not remember what type and color of taxi they rode in or where the robbery happened, the police official said. The swimmers also could not say what time the events occurred.

Later in the day, Lochte described the incident to NBC's "Today" show.

"We got pulled over, in the taxi, and these guys came out with a badge, a police badge, no lights, no nothing, just a police badge, and they pulled us over," Lochte said. "They pulled out their guns, they told the other swimmers to get down on the ground -- they got down on the ground. I refused, I was like, 'We didn't do anything wrong, so -- I'm not getting down on the ground.'

"And then the guy pulled out his gun, he cocked it, put it to my forehead and he said, 'Get down,' and I put my hands up, I was like, 'Whatever.' He took our money, he took my wallet -- he left my cellphone, he left my credentials."

Word of the robbery initially created confusion between Olympic and U.S. officials. An International Olympic Committee spokesman at first said reports of the robbery were "absolutely not true," then reversed himself, apologized and said he was relying on initial information from the USOC that was wrong.

Lochte told USA Today Sports that he and his teammates didn't initially tell the U.S. Olympic officials about the robbery "because we were afraid we'd get in trouble."

Information from ESPN's Wayne Drehs and The Associated Press was used in this report.