Former White House communications director Anthony Scaramucci says President Trump should have come out harder against white supremacy after Saturday's violence in Charlottesville, Virginia.

"I think he needed to be much harsher as it related to the white supremacists and the nature of that," Scaramucci said during an interview Sunday on ABC's "This Week."

He said that if he was still working in the White House, he would not have recommended Trump make the statement he did.

During the interview, his first sitdown TV interview since he left the White House after 10 days on the job, Scaramucci also told ABC he believed chief strategist Steve Bannon was one of the White House's biggest problems.

"He has to move away from that sort of 'Bannon-bart' nonsense," Scaramucci said, apparently combining Bannon's name with Breitbart, the website that Bannon ran before joining Trump's campaign last year.

"The whole thing is nonsensical," Scaramucci said. "It's not serving the president's interests."

The White House did not immediately respond to CNNMoney's request for comment about Scaramucci's remarks.

Breitbart has close ties with the alt-right movement, an ideology that can be associated with white nationalism. Bannon has described Breitbart as "the platform for the alt-right."

Scaramucci's remarks came a day after a rally in Virginia turned violent, as clashes broke out between white nationalists and counterprotesters. One person died when a car plowed into a crowd.

Trump's response was criticized by politicians on both sides of the aisle for failing to name white supremacist or alt-right groups involved in the protests. Instead Trump condemned the "egregious display of hatred, bigotry and violence on many sides, on many sides."

The White House on Sunday released a statement that said "of course" Trump condemns "white supremacists, KKK, neo-nazi and all extremist groups."

-- CNNMoney's Tom Kludt and CNN's Dan Merica contributed to this report.