President Trump should have been “harsher” on the white nationalists who organized the deadly rally this weekend in Virginia, Anthony Scaramucci said Sunday, in his first TV interview since being fired as White House communications director.

"I wouldn't have recommended that statement. I think he would have needed to have been much harsher. … With the moral authority of the presidency, you have to call that stuff out," Scaramucci told ABC’s “This Week” about Trump’s comments after the deadly protests and counter-protests Saturday in Charlottesville, Va., surrounding the “Unite the Right” rally.

Trump gave Scaramucci the White House job in late July, in part for how he deftly forced CNN to take down an incorrect online story connecting him to the Russia investigations.

However, he was fired 11 days later, after a profanity-laced phone call with The New Yorker reporter Ryan Lizza and amid the arrival of retired Marine Gen. John Kelly as the new White House chief of staff.

“For the record, I thought that (call) was off the record,” Scaramucci, a former Wall Street financier and member of the Trump campaign’s finance committee, said Sunday. “That was a very deceitful thing that he did. … But we don't need to debate that anymore. That's past news. I made a mistake. I'm accountable for the mistake. I paid the consequences of that.”

He also suggested that White House chief strategist Steve Bannon is the source of at least some of the West Wing leaks and that his connection to Breitbart.com and its association with the far right is hurting the presidency.

“It's not serving the president's interests,” Scaramucci told ABC News. “He's got to move more into the mainstream. He's got to be more into where the moderates are and the independents are. … If he does that, he'll have a very successful legislative agenda. … And if he doesn't do that, you're going to see inertia and you're going to see this resistance from more of the establishment senators that he needs to curry favor with.”

Still, Scaramucci made clear that Bannon’s future in the White House is solely in the hands of Trump.

"I think the president knows what he's going to do with Steve Bannon,” he said. “It's his decision. But at the end of the day, the president has a very good idea of who the leakers are inside the White House. The president has a very good idea of the people who are undermining his agenda that are serving their own interests."