CARTAGENA, Colombia — Twelve Secret Service agents assigned to this port city in advance of President Obama’s arrival for the Summit of the Americas were relieved of duty, amid accusations of misconduct that is said to have involved prostitution, a government official confirmed Friday.

The misconduct is said to have occurred before Mr. Obama arrived here on Friday afternoon for the meeting of more than 30 world leaders that begins Saturday.

White House officials would not comment, referring all questions to the Secret Service. News of the accusations was first reported by The Washington Post.

In a statement, a Secret Service spokesman, Edwin M. Donovan, acknowledged that agents had been recalled because of accusations of misconduct and that they would be replaced by other Secret Service personnel, but he did not address the issue of prostitution.