Wayne Drehs says Ryan Lochte's legacy will be forever changed after Brazilian authorities accused him of fabricating a story about being robbed at gunpoint. (1:50)

A Brazilian police official has told The Associated Press that American swimmer Ryan Lochte fabricated a story about being robbed at gunpoint in Rio de Janeiro.

The official said Lochte's teammates Jack Conger and Gunnar Bentz, who were pulled off a plane going back to the United States late Wednesday, told police that the robbery story had been fabricated.

The official, who has direct knowledge of the investigation, spoke on the condition of anonymity because he was not authorized to speak about an ongoing investigation.

He said that around 6 a.m. Sunday, Lochte, along with Conger, Bentz and Jimmy Feigen, stopped at a gas station in Barra da Tijuca, a suburb of Rio where many Olympic venues are located. One of the swimmers tried to open the door of an outside bathroom. It was locked.

A few of the swimmers then pushed on the door and broke it. A security guard appeared and confronted them, the official said. The guard was armed with a pistol, the official said, but he never took it out or pointed it at the swimmers.

According to the official, the gas station manager then arrived. Using a customer to translate, the manager asked the swimmers to pay for the broken door. After a discussion, they paid him an unknown amount of money and left.

Earlier Thursday, authorities told ABC News there is video of the gas station incident. ABC News also reports that the swimmers offered 100 Brazilian reals ($31) and $20 to compensate for the damage.

"Unfortunately, the swimmers told one lie after another," a Brazilian police official who had reviewed the video images told The New York Times. "We've been able to determine that there was no armed assault."

Lochte first lied about the robbery to his mother, Ileana Lochte, who spoke with reporters, the police official told the AP. That led to news coverage of the incident and prompted police attention.

A police news conference to discuss the investigation is slated for 2 p.m. ET Thursday at DEAT, a precinct that is part of the civil police in Rio de Janeiro.

Conger and Bentz arrived at DEAT around 12:30 p.m. ET Thursday for questioning.

International Olympic Committee spokesman Mario Andrada said during a news conference Thursday that neither Lochte nor his teammates need to apologize for the incident.

"We have to understand that these kids came here to have fun. Let's give these kids a break," Andrada said. "Sometimes you make decisions you later regret. They had fun. They made a mistake. Life goes on."

Andrada's comments were made before the reports that a robbery did not happen.

Police say Ryan Lochte made up his story about being robbed following an incident in which he and three teammates damaged a gas station bathroom, according to multiple reports. Matt Hazlett/Getty Images

Conger, Bentz and Feigen remain in Brazil, as authorities have ordered the seizure of their passports. A Brazilian judge also ordered Lochte's passport be seized, but he said he has already returned to the U.S.

Brazilian authorities said in a statement Thursday that Conger and Bentz were taken off the airplane. A police source told ESPN Brasil that the swimmers were interviewed for just more than three hours at the airport, with a Brazilian lawyer and representatives from the USOC and the U.S. Consulate present.

Stephen Conger, the father of Jack Conger, told ESPN on Thursday that his son slept in a hotel Wednesday night and was guarded by U.S. government personnel. Jack was never under arrest, according to his father. "It's more a political ordeal than a criminal one," he said.

Stephen Conger said the plan is for Jack to be on a plane home Thursday night. He said he's shared text messages with his son, but said "We're as in the dark as anyone about what happened. I don't know if there was a robbery or not."

NBC spoke with Lochte on Wednesday night and reported that he said he had left Rio and already returned home, as originally scheduled.

Lochte backtracked on his initial comments to the network made Sunday, saying the barrel of the gun he had said was placed to his forehead was instead pointed in his direction. He also said the taxi wasn't pulled over but that the swimmers were robbed after making a stop at a gas station.

He said when he was interviewed by police about the incident that no one had suggested he needed to stay in Brazil, nor had any officials expressed doubts over his account of the robbery, NBC reported.

Brazilian authorities continued pressing the American swimmers over the ever-changing account of the robbery. Police said the swimmers were unable to provide key details in early interviews and they found little evidence to support the robbery claim. The swimmers said they had been intoxicated and could not remember what type and color of taxi they rode, where the incident happened or what time the events occurred, police said. The police official said officers grew suspicious when they reviewed security video of the swimmers returning to the athletes village and saw them wearing watches.

The office of Judge Keyla Blanc, who ordered the passports seized, said there were discrepancies in the statements by the swimmers.

Lochte told USA Today that he and his teammates didn't initially tell U.S. Olympic officials about the robbery because "we were afraid we'd get in trouble."

Information from ESPN's Shaun Assael and ESPN Brasil's Gabriela Moreira was used in this report.