President Obama said Sunday that he wants the investigation of the prostitution scandal that led 11 U.S. Secret Service agents to be returned home from Cartagena, Colombia, where they had been sent to provide protection for him, to be thorough and rigorous.

During a news conference Sunday at the conclusion of the Summit of the Americas, Obama said he expects all agents to conduct themselves with “dignity and probity” and all U.S. personnel to “observe the highest standards” when serving abroad.

If the allegations that the men brought prostitutes to their hotel rooms on Wednesday prove true, Obama said, “then, of course, I’ll be angry.”

“We’re representing the people of the United States,” he said.

Obama’s remarks were his first about the growing scandal. The Secret Service placed the men on administrative leave pending an internal investigation. The agents — who were removed from Cartagena on Thursday and replaced with a new team shortly before Obama’s arrival Friday — were returned to Washington and interviewed by the Office of Professional Responsibility, the agency’s internal affairs unit.

Meanwhile, the Defense Department has ordered its own inquiry after determining that five of its personnel, who were staying at the same hotel as the Secret Service agents, violated curfew on Wednesday night. All of the U.S. personnel were part of Obama’s advance team that was preparing logistics and security for his arrival.

The alleged misconduct came to light after one of the agents became involved in a dispute with a woman Thursday morning over a payment, and Colombian police reported the matter to the U.S. Embassy.

The controversy has shifted some attention away from Obama’s trip to the economic summit, at least in the United States, where the media have focused on the accusations of heavy drinking and womanizing.

Obama was asked about the matter by a reporter in his joint news conference with Colombian President Juan Manuel Santos. Obama made a point to praise the Secret Service in general, emphasizing that the agency does “very hard work under very stressful circumstances.”

“I’m very grateful for what they do,” he said. “I will wait until the full investigation [is completed] until I pass final judgment.”

Lawmakers in Washington were pressing Sunday for more details about the investigation.

House Oversight and Government Reform Committee Chairman Darrell Issa (R-Calif.), whose panel maintains jurisdiction over all federal agencies, said he has reason to believe that more than 11 agents were involved.

“We think the number might be higher, and we’re asking for the exact amount of all the people who, quote, were involved,” he said on CBS’s “Face the Nation.”

Issa did not elaborate on that statement, and his office did not respond to follow-up questions from The Washington Post.

On the talk show, Issa also said: “This kind of a breach is a breach in the federal workforce’s most elite protective unit, and they don’t just protect the president, of course; they protect the Cabinet members, the vice president, the first family, candidates. So when you look at this, you realize if you can have this kind of breakdown, one that could lead to blackmail . . . then we’ve got to ask: Where are the systems in place to prevent this in the future?”

The Secret Service agents allegedly involved are a mix of special agents, who provide personal protection for the president and other high-level officials, and members of the uniformed division, who handle logistical support and building security, officials briefed on the investigation have said.

Jon Adler, president of the Federal Law Enforcement Officers Association, said a lawyer representing the agents did not want to discuss the matter with reporters until it had been adjudicated.

Staff writer Joe Davidson contributed to this report.