Washington (CNN) As pressure mounts around the country demanding the removal of monuments lionizing the Confederacy, President Donald Trump said Tuesday that the removal of such monuments is "changing history."

"I wonder, is it George Washington next week?" Trump asked.

Trump, in an agitated back-and-forth with reporters in Trump Tower, defended members of the "Unite the Right" protest last weekend in Charlottesville, Virginia, where a white supremacist has been charged with murder for ramming a car into a crowd of counter-protesters. He went on to make a slippery slope argument -- equating Confederate general Robert E. Lee with presidents like Washington and Thomas Jefferson, who were slave owners.

Trump said while he condemned the neo-Nazi elements of the protest, some protesters were there for acceptable purposes.

"I've condemned neo-Nazis. I've condemned many different groups. But not all of those people were neo-Nazis, believe me," Trump said. "Not all of those people were white supremacists, by any stretch. Those people were also there because they wanted to protest the taking down of a statue of Robert E. Lee."

Charlottesville, like many southern towns across the US, has grappled with its Confederate legacy and the decision in the past to honor men who fought to keep the institution of slavery alive.

Earlier this year, the city of New Orleans removed several of its Confederate memorials, marking the end of the project by bringing down a statue of Robert E. Lee in May. New Orleans Mayor Mitch Landrieu said in a speech about the monument removal that they had served to gloss over the Confederacy's legacy instead of truly reflecting history.

"To literally put the Confederacy on a pedestal in our more prominent places -- in honor -- is an inaccurate recitation of our full past, is an affront to our present and it is a bad prescription for our future," Landrieu said.

The fast change in the historic southern city was met with a great deal of criticism, and the clash between those who support the symbols erected to honor the Confederate legacy and those who want them down has generated repeated flashpoints.

During a Monday protest in Durham, North Carolina, a woman toppled a Confederate statue, and authorities said on Tuesday she was arrested.

As the fight over the monuments and what they signify continues, other monuments across the country face removal.

In his comments, Trump lamented the push to take down Confederate statues.

"So, this week it's Robert E. Lee," Trump said. "I notice that Stonewall Jackson is coming down. I wonder is it George Washington next week, and is it Thomas Jefferson the week after? You know, you really do have to ask yourself, where does it stop?"

Trump declined, however, to say he supported keeping or removing the statue, and said such decisions should be left "to a local town community or the federal government depending on where it is located."

Trump appeared incredulous at the prospect of removing the statues, and implied that removing Confederate statues was akin to removing monuments for Washington and Jefferson, both men who championed the cause of liberty for the US against the British while they also owned slaves.

"George Washington was a slave owner. Was George Washington a slave owner? So will George Washington now lose his status?" Trump said. "Are we going to take down statues to George Washington? How about Thomas Jefferson? ... Are we going to take down the statue? Because he was a major slave owner. Now are we going to take down his statue?"

He added, "You're changing history. You're changing culture."

Over the weekend, right-wing white supremacist protesters gathered in the Virginia city, bearing tiki torches and in some cases chanting racist slogans. Counter-protesters gathered in response, and the two groups clashed. A man drove a car into a group of counter-protesters Saturday, killing one and injuring more than a dozen others.

Trump initially responded to the violent, racially charged events with a statement casting blame "on many sides." After significant criticism for equating the two sides, Trump issued a statement Monday condemning neo-Nazis and white supremacists.