WASHINGTON -- The leaders of four minority House caucus groups have written a letter to President Donald Trump calling for the removal of White House staff aides Steve Bannon, Stephen Miller and Sebastian Gorka.

The heads of the black, Hispanic, Asian and progressive caucuses are calling in the letter for the firings of the Trump administration officials in the wake of a violent, racist rally in Charlottesville, Virginia. The letter asserts their continuing presence in the White House is emboldening a resurgent white supremacist movement in America.

"Americans deserve to know that white nationalists, white supremacists, and neo-Nazis are not in a position to influence U.S. policy," says the letter dated Monday. "In this time of tumult in our country, Americans deserve a leader that will bring us all together and denounce those who seek to tear us apart."

Get Breaking News Delivered to Your Inbox

Bannon's position as Mr Trump's chief strategist may be in jeopardy, two sources close to the White House told CBS News' chief White House correspondent Major Garrett on Monday. There is a growing feeling both inside and outside the White House that Bannon is increasingly falling out of favor with Mr. Trump, with the relationship between the two falling to all-time lows.

One source says that Bannon could be gone by the end of the week, Garrett reports.

Over the weekend, White House officials were not quick to come to Bannon's defense on the subject of his loyalty to the president and the White House's agenda.