President Trump's planned statements on infrastructure derailed into explosive comments about the violence in Charlottesville, Virginia, in which the president defended his Saturday statement that there was hatred and violence, "on many sides."

Mr. Trump, speaking from a podium at Trump Tower on Tuesday, said the "alt-left" bears some responsibility for the violence in Charlottesville, and "nobody wants to say that," blaming violence on "both sides." The speech was widely criticized by Democrats.

"Let me ask you this," Mr. Trump said. "What about the fact that they came charging, that they came charging with clubs in their hands, swinging clubs. Do they have any problem? I think they do."

Get Breaking News Delivered to Your Inbox

"I will tell you something," the president added. "I watched this very closely, much more closely than you people watched it and you have, you had a group on one side that was bad and you had a group on the other side that was also very violent. And nobody wants to say that, but I'll say it right now. You had a group on the other side that came charging in without a permit and they were very, very violent."

Mr. Trump was accused -- even by top members of his own party -- of failing to call out the white supremacists at a rally that left one woman dead after a suspected attacker rammed his car into a sea of people. Mr. Trump was pressured to renounce the Nazi and white supremacist supporters behind Saturday's deadly events, and eventually did so on Monday.

But on Tuesday, Mr. Trump circled back to his earlier comments, defending his original statements by saying he wanted to be sure he had all the facts.

"I didn't wait long," Mr. Trump said. "I didn't wait long. I didn't wait long. I wanted to make sure, unlike most politicians, that what I said was correct. Not make a quick statement. The statement i made on Saturday, a quick statement, the first statement, was a fine statement. But you don't make statements that direct to get the facts. It takes a little while to get the facts."

The president did condemn the suspected driver, James Alex Fields Jr., calling him a, "murderer." Mr. Trump has not yet reached out to the family of Heather Hayer, the woman killed in the apparent attack.

"The driver of the car is a murderer, and what he did is a horrible, horrible inexcusable thing," Mr. Trump said.

But he couldn't resist revisiting the comments he made Saturday.

"I think there's blame on both sides," Mr. Trump said.

CBS News White House and senior foreign affairs correspondent Margaret Brennan reports new chief of staff John Kelly was standing off to the sidelines looking at shoes during Mr. Trump's remarks. Kelly, a retired four-star Marine general, was brought onto Mr. Trump's staff to establish command and control within the White House. But the limits of his ability to keep the president on message were laid bare Tuesday, Brennan reports.

Mr. Trump's remarks were only supposed to be brief ones on infrastructure.

The president said he feared the movement to remove controversial statutes was going too far, asking if people would support the removal of statues depicting former presidents George Washington or Thomas Jefferson. The Charlottesville "Unite the Right" rally was originally set up to protest the removal of a statute of Confederate general Robert E. Lee from a public park.

"Those people, all of those people, excuse me, I've condemned neo-Nazis, I've condemned many different groups," Mr. Trump said. "But not all those people were Neo-nazis, believe me. Not all those people were white supremacists. Those people were also there because they wanted to protest the taking down of a statue Robert E. Lee.

So, excuse me, when you take a look of some of the groups and you see, and you'd know if if you were honest reporters and many of you are not, many of those people were there to protest the taking down of the statue of Robert E. Lee. So this week it's Robert E. Lee, I notice Stonewall Jackson is coming down. I wonder is it George Washington next week and is it Thomas Jefferson the week after? You really do have to wonder, where does it stop?"

President Trump answers questions about his responses to the violence, injuries and deaths at the "Unite the Right" rally in Charlottesville as he talks to the media in the lobby of Trump Tower in New York on Tue., Aug. 15, 2017. Reuters

The president also made remarkable comments as he was leaving the podium, saying he owns a large "house" in Charlottesville. When a reporter asked if he meant his winery, he said yes.

"I mean, I know a lot about Charlottesville," Mr. Trump said. "Charlottesville is a great place that's been very badly hurt over the last couple of days. I own actually one of the largest wineries in the United States. It's in Charlottesville."

Elaine Chao, the transportation secretary, and Gary Cohn, the president's chief economic adviser, were forced to awkwardly come to the podium to take questions from reporters on infrastructure after Mr. Trump's explosive, off-the-cuff remarks.

Mr. Trump is spending the day at New York City's Trump Tower, in the middle of a two-week working vacation mostly spent in Bedminster, New Jersey.