Washington (CNN) A coalition of major civil rights and faith groups on Sunday called on President Donald Trump to "directly disavow the white supremacists" who participated in violent protests in Charlottesville, Virginia, this weekend -- a reference to Trump's remarks condemning the deadly clashes on Saturday.

"It represents a failure of leadership from the nation's chief executive," the Leadership Conference on Civil and Human Rights, an umbrella group, said in a statement. "It is long past time for Trump to personally and unequivocally denounce white supremacy, violent extremism, and hate in all its forms."

The group also called for the ouster of White House chief strategist Steve Bannon and deputy assistant Sebastian Gorka, who have drawn criticism from civil rights organizations for their associations with the alt-right, a hodgepodge of far-right, white nationalist groups that drafted off the President's 2016 campaign to rise to national prominence.

Vanita Gupta, the Leadership Conference's president and CEO, said on a conference call with reporters to discuss her group's demands that supporters of white supremacists "should not serve in the White House or in any level of government."

Farhana Khera, the president and executive director of Muslim Advocates, said that Bannon and Gorka's presence in the administration sends the "silent message to extremists to continue their march of terror."

The White House did not return a request for comment Sunday night.

Before becoming Trump's campaign chief last summer, Bannon ran the conservative news site Breitbart, which he called "the platform for the alt-right." Meanwhile, Gorka, a former Breitbart national security editor, has been outspoken on the need to confront Islamist terrorism, drawing fire last week week when he spoke with the news site about terrorism and said white supremacists were not the problem.

"It's this constant, 'Oh, it's the white man. It's the white supremacists. That's the problem,'" Gorka said. "No, it isn't."

Since his remarks Saturday, Trump has remained silent on the chaos that gripped Charlottesville over the weekend, which left three dead and dozens injured after white supremacists and counterprotesters clashed in the streets. But senior White House aides, including homeland security assistant Tom Bossert, sought to clarify Trump's comments that "many sides" were to blame for the violence in appearances on talk shows Sunday morning.

When asked on CNN's "State of the Union" about Gorka's comment to Breitbart, Bossert said it was "certainly not" the position of the White House that white supremacy was not a problem and added he was not certain of the context of Gorka's quote.

"I condemn white supremacists and racists and white Nazi groups and all the other groups that espouse this kind of hatred," Bossert also said when pressed on the President's position towards the white supremacists.

A White House official, who requested anonymity and ignored attempts to go on the record, told reporters Sunday that it was obvious the President condemned "white supremacists, KKK, neo-Nazi and all extremist groups" despite Trump not mentioning those groups during his remarks at his private golf club in Bedminster, New Jersey, on Saturday; Trump instead blamed the violence in Charlottesville on "many sides."

"The President said very strongly in his statement yesterday that he condemns all forms of violence, bigotry and hatred, and, of course, that includes white supremacists, KKK, neo-Nazi and all extremist groups," the official said. "He called for national unity and bringing all Americans together."

The Leadership Conference's statement called Trump's response to the unrest in Charlottesville "weak, insufficient and unacceptable." Gupta demanded the President personally disavow the white supremacists and call the violence over the weekend "domestic terrorism." The group added that Bannon and Gorka were responsible for stoking hate and division.

Richard Cohen, the president and CEO of the Southern Poverty Law Center, echoed those statements during the conference call.

"President Trump should apologize for what he has done, and he should follow up his apology with concrete actions to un-do the horrible damage that his campaign, his rhetoric and his policies have caused," Cohen said. "Firing Mr. Bannon and Mr. Gorka would be a sign that he is serious about changing direction."

Gupta also called for Congress to hold a hearing with the departments of Justice and Homeland Security to address investigate, and root out hate crimes.

The NAACP Legal Defense and Educational Fund, Anti-Defamation League, Human Rights Campaign, and the National Council of Churches also participated in the conference call.

In a separate statement Sunday, the United States Holocaust Memorial Museum condemned the "violence and neo-Nazi, racist and anti-Semitic symbols and language used by some of the participants" in the protests, which were sparked by the city's plans to remove a statue of Confederate Gen. Robert E. Lee from a downtown park. The museum also said neo-Nazism in any form is "antithetical to Americans values and has no place in American society."