President Trump caught a lot of heat this week when he asked whether statues of George Washington or Thomas Jefferson would be next if those who wished to take down a statue of Robert E. Lee in Charlottesville, Va., got their way.

But he could have backed up his remark by pointing to something that happened here in New York last fall when demonstrators demanded that the well-known statue of one of his predecessors — Theodore Roosevelt — be taken down from its familiar perch in front of the Museum of Natural History.

The problem with Trump’s statement wasn’t so much the rights and wrongs of the statue controversy. The context was an attempt by neo-Nazis and the Ku Klux Klan to use Lee as a rallying point for an anti-Semitic and racist demonstration that led to a vehicular terror attack. The president should have stuck to a condemnation of those horrors rather than engaging in moral equivalence between the racists and their opponents or the debate over statues.

But once we get past Trump’s stubborn refusal to stick to a position that might unite the country, the question of toppling statues must be addressed. Public images of America’s past are quickly becoming the focus of revisionist history that some liberals believe requires us to tear down memorials to those associated with slavery, or, as in the case of TR, prone to say some things that would be judged politically incorrect in the 21st century.

That iconic equestrian image of TR (in which he’s shown as a cowboy rather than as a Spanish American War Rough Rider as the “Night at the Museum” movie series depicted the statue) drew fire because protesters see the monument, in which the 26th president is flanked by an African-American and Native American, as an expression of white supremacy.

Many on the left also damn Roosevelt as an imperialist, although without his efforts to establish America as a global power, it is unlikely that the United States could have saved the world from the Nazis a generation later.

Are revisionist radicals capable of enforcing a new standard that could lead to past presidents being evicted from our public squares? Maybe.

A generation ago, nobody would have thought Washington or Jefferson controversial. Now it appears that even Mount Rushmore isn’t safe, since among its quartet of greats, only Lincoln might be exempt from the iconoclasts.

The movement to send America’s greatest heroes down George Orwell’s “memory hole” will likely gain more support now that neo-Nazis and, perhaps, Trump are seen as the primary advocates of maintaining the status quo.

But, contrary to Trump’s assertion, it doesn’t have to be a simple choice of keep them all or tear them all down.

While many of America’s Founding Fathers owned slaves, there’s a clear difference between their hypocrisy and those who sought to destroy the republic they created in order to safeguard slavery.

While we honor Washington and Jefferson for their defense of freedom, symbols like the Confederate battle flag and many of the monuments to that lost cause are more about the defense of Jim Crow than the memory of the Civil War.

Even among Confederate statuary, there are distinctions to be made. Surely, memorials honoring fallen soldiers rather than the cause they served — such as the memorial in Durham, NC, that was torn down by a leftist mob this week — should be exempt. The problem now is that if statues of figures like Lee are to become rallying points for violent hate groups, then there’s an argument to be made for relegating them to museums and out of public view.

Yet we also have a duty to defend figures like Roosevelt from these depredations. In addition to being a great president and uniquely heroic figure, TR’s place in front of the museum on Central Park West is a tribute to his status as our nation’s greatest environmentalist and has nothing to do with contemporary notions about diversity.

It’s possible and necessary for a rational society to make distinctions between monuments erected to bolster segregation and those that are genuine expressions of patriotism and America’s highest ideals.

Those who lack the moral compass to tell the difference shouldn’t be making these kinds of decisions.

Jonathan S. Tobin is opinion editor of JNS.org and a contributor to National Review.