The US secret service says it has put 11 agents on leave while it investigates alleged misconduct before a summit attended by Barack Obama in Colombia, as it emerged that five soldiers are also facing investigation.

The secret service apologised for the distraction the incident had caused at the summit in Cartagena, where Barack Obama is meeting leaders of more than 30 countries.

The US Southern Command announced that five service members, who were assigned to support the secret service and were staying at the same hotel as the agents under investigation, had violated their curfew and may have been involved in inappropriate conduct. The soldiers were confined to their quarters in Colombia and ordered not to have contact with others.

All the alleged activities took place before Obama arrived at the summit on Friday.

The misconduct being investigated involved prostitutes, it has been confirmed. Peter King, chairman of the House homeland security committee, said after he was briefed on the investigation that "close to" all 11 of the agents involved had brought women back to their rooms at a hotel separate from the one where Obama is staying.

The New York Republican representative said the women were "presumed to be prostitutes" but investigators were interviewing the agents.

King said he was told that anyone visiting the hotel overnight was required to leave identification at the front desk and leave the hotel by 7am. When a woman failed to do so, it raised questions among hotel staff and police, who investigated. They found the woman with the agent in the hotel room and a dispute arose over whether the agent should have paid her.

King said he was told that the agent did eventually pay the woman.

The incident was reported to the US embassy, prompting further investigation, King said.

The White House said Obama had been briefed about the incidents but would not comment on his reaction.

"The president does have full confidence in the United States secret service," the presidential spokesman Jay Carney said when asked.

Carney insisted the matter was more a distraction for the media than Obama. But the secret service assistant director Paul Morrissey said in a statement: "We regret any distraction from the Summit of the Americas this situation has caused."

He said the 11 employees in question were special agents and uniformed division officers, and none were assigned to directly protect Obama. All were sent home and replaced, Morrissey said, given "the nature of the allegations" and a zero tolerance policy on personal misconduct.

The secret service said the incidents had had no bearing on its ability to provide security for Obama.

General Douglas Fraser, the commander of US Southern Command, said he was "disappointed by the entire incident" and the behaviour was "not in keeping with the professional standards expected of members of the United States military".

Colonel Scott Malcom, chief of public affairs for Southern Command, confirmed that the five soldiers had violated a curfew. He could not confirm whether service members were involved in the alleged prostitution scandal.

The agents and soldiers were at Cartagena's five-star Hotel Caribe, where several members of the White House staff and press corps also later stayed.

A hotel employee said the agents had arrived at the beachfront hotel about a week ago and left on Thursday, a day before Obama and other regional leaders arrived for the weekend summit. Three waiters described the agents as drinking heavily during their stay.