Washington (CNN) Charlottesville Democratic mayor Michael Signer said Sunday that President Donald Trump had emboldened organized racists who caused the racially charged violence in his city over the weekend.

"Look at the campaign he ran," Signer said on CNN's "State of the Union."

Signer called on everyone to join together to recognize and combat racism, adding that the effort should include Trump.

White supremacist demonstrators clashed with counterprotesters in the Virginia city on Saturday, with the violence culminating with a man ramming a car through a group of counterprotesters. Authorities said the attack injured 19 and killed one.

Signer accused Trump of running a campaign that attracted racist groups and said he had personally opted not to push back forcefully against them.

"Look at the intentional courting both, on the one hand, of all these white supremacists, white nationalist groups like that, anti-Semitic groups," Signer said. "And then look on the other hand, the repeated failure to step up, condemn, denounce, silence, you know, put to bed all those different efforts."

However, Signer said the country would move ahead, and the problems went beyond what he saw in Trump.

"To be honest, this is not about Donald Trump," Signer said, adding that the violence from white supremacists demonstrated that the country's core democratic principles were eroding, but people had an opportunity to change for the better.

"People will react to the darkness with a whole lot more light," he said.

Signer called the racist demonstrators, marching with tiki torches in hand, a "visible display of intimidation" and said he expected an aggressive case to be made against the man behind the deadly attack.

"I hope that if the facts are there, that we vigorously prosecute this as a case of domestic terrorism," Signer said. "This cannot be tolerated."

White House reaction

On the same program, White House Homeland Security adviser Tom Bossert defended Trump's remarks, saying the President's statement did not equivocate the white supremacists with the counterprotesters.

Bossert blamed "both sides" for Saturday's violence, while also saying the White House was "absolutely" against white supremacists.

"I'm sure there were good people in the groups that had various opinions on the removal or maintenance of the statue," Bossert said. "But what they found when they showed up were groups from outside that showed up on both sides, looking for trouble, dressed in riot gear, prepared for violence."

Bossert added that both he and Trump wanted the attacker to face "swift justice" and noted the Department of Justice was pursuing a civil rights investigation.

"This individual should face swift justice," Bossert said. "The President of the United States shares that view. I know he does."

And he went on to say again he took issue with the groups on both sides.

"These groups showed up spewing hate," Bossert said. "These groups showed up looking for violence."

Asked what groups he was referring to, Bossert replied, "I refer to the groups that clashed yesterday."

Pressed again on the President's position towards the white supremacists, Bossert offered a condemnation of hate groups.

"I condemn white supremacists and racists and white Nazi groups and all the other groups that espouse this kind of hatred," he said.

Trump has faced regular criticism over some of the people he chose as his top aides, including chief strategist Steve Bannon -- who previously ran the conservative news site Breitbart, which Bannon called "the platform for the alt-right" -- and deputy assistant Sebastian Gorka, who drew fire earlier in the week when he spoke with Breitbart about terrorism and said white supremacists were not the problem.

"It's this constant, 'Oh, it's the white man. It's the white supremacists. That's the problem,'" Gorka said. "No, it isn't."

When asked about the comment on CNN's "State of the Union," Bossert said it was "certainly not" the position of the White House that white supremacy was not a problem and added he was not certain of the context of Gorka's quote.