The debate over Confederate monuments has been framed by President Donald Trump — and some who share his views — as a fight between those who wish to preserve history and those who would “erase” it. But let us linger on what history we’ll be preserving as long as Confederate memorials stand.

The Confederate monuments in New Orleans; Charlottesville, Virginia; Durham, North Carolina, and elsewhere did not organically pop up like mushrooms. The installation of the 1,000-plus memorials across the US was the result of the orchestrated efforts of white Southerners and a few Northerners with clear political objectives: They tended to be erected at times when the South was fighting to resist political rights for black citizens. The preservation of these monuments has likewise reflected a clear political agenda.

It is going to take equal energy and focus to remove them from the national landscape.

But the story of the monuments is even stranger than many people realize. Few if any of the monuments went through any of the approval procedures that we now commonly apply to public art. Typically, groups like the United Daughters of the Confederacy (UDC), which claimed to represent local community sentiment (whether they did or did not), funded, erected, and dedicated the monuments. As a consequence, contemporaries, especially African Americans, who objected to the erection of monuments had no realistic opportunity to voice their opposition.

Most Confederate monuments were, in short, the result of private groups colonizing public space.

Over the past decade, Southern legislatures have passed laws requiring approval from state legislatures before any historical monuments can be moved, removed, or altered — thereby freezing those private decisions in place.

A controversy in Reidsville, North Carolina in 2011, which failed to attract any national attention, offers a window into the origins of Confederate monuments and their contested “ownership.” That year, an errant driver plowed into the generic Confederate soldier memorial that stood precariously beside a major street in the small town, 25 miles north of Greensboro.

Because other motorists had previously hit the monument, the UDC, which had funded and erected the monument in 1910, decided the sculpture would be safer if it was moved to a nearby cemetery. But in a strange twist, the plan was blocked when the Sons of Confederate Veterans, another Confederate heritage organization, sued the UDC to prevent the relocation of the monument. Eventually, the UDC prevailed and the restored monument was rededicated in the cemetery in 2014. The city itself was a spectator in this legal fight.

Had the dispute flared after 2015, when the state legislature passed a law effectively blocking the removal of monuments, the UDC would have had to tangle not only with neo-Confederates but also with state legislators.

A smaller number of monuments, like the one recently toppled in Durham, were indeed funded with public money — but an asterisk must be attached to the word “public.” In 1922, Confederate veterans in Durham persuaded the state legislature to allocate $5,000 of county taxes to fund the monument. No one asked black residents, who were denied the right to vote by Jim Crow laws, whether they supported spending their tax dollars on this public, political statement.

Most monuments went up not immediately after the war, but as Southerners put Jim Crow in place — and Northerners gave up on racial justice

Let us acknowledge that the architectural landscapes we have inherited are neither sacred nor unchanging. The timing of the proliferation of the monuments themselves illustrates this point. In the years immediately after the Civil War, North Carolina Confederates understandably mourned their dead, yet the state erected fewer than 30 memorials between 1865 and 1890. Then, during the next half century, they dedicated more than 130.

It is hardly coincidence that the cluttering of the state’s landscape with Confederate monuments coincided with two major national cultural projects: first, the “reconciliation” of the North and the South, and second, the imposition of Jim Crow and white supremacy in the South. As part of the process of national reconciliation, white Northerners agreed to tolerate the commemoration of Confederates, and they contributed both moral support and funds to the veneration of a few Confederate figures in particular, especially Robert E. Lee.

Lee became a convenient icon of reconciliation who was depicted as having reluctantly fought to protect his native state — not slavery— and then after the war devoted himself to the uplift of the South and to binding the nation’s wounds. For white Northerners, Lee was a military hero who could be venerated without having to embrace the Confederate cause in its totality. (This impulse explains the monuments to Lee in the US Capitol, at the City University of New York, and other sites outside of the former Confederacy.)

Meanwhile, white Southerners used the commemoration of the Confederacy to promote a degree of white cultural unity that had never existed in the region either before or during the Civil War. An observer scanning the commemorative landscape of North Carolina will see little evidence of the tens of thousands of white North Carolinians who fought for the Union, the even larger number of white North Carolinians who actively opposed the Confederacy, or the tens of thousands of African Americans who escaped slavery and joined the Union army.

Confederate commemorators suppressed these unwelcome blemishes to their preferred version of history while simultaneously making the Confederate cause virtually sacred. White Southerners who questioned the Confederate narrative faced ostracism or worse.

Some contemporaries linked the monuments to the defense of white supremacy in shockingly explicit terms

The pursuit of white cultural unity through Confederate commemoration went hand-in-hand with the promotion of white supremacy. The Confederate monuments themselves were sometimes explicitly linked to the cause of white supremacy by the notables who spoke at their dedication. For instance, at the 1913 dedication of an on-campus monument honoring University of North Carolina students who fought for the Confederacy, white industrialist Julian Carr unambiguously urged his audience to devote themselves to the maintenance of white supremacy with the same vigor that their Confederate ancestors had defended slavery.

During the dedication speech, Carr praised Confederate soldiers not just for their wartime valor but also for their defense “of the Anglo Saxon race during the four years after the war” when “their courage and steadfastness saved the very life of the Anglo Saxon race in the South.” The “four years after the war” was a clear reference to the period in which the Ku Klux Klan, a white paramilitary organization terrorized blacks and white Republicans who threatened the traditional white hierarchy in the state. Then he boasted that “one hundred yards from where we stand” — and within months of Lee’s 1865 surrender — “I horse whipped a negro wench until her skirts hung in shreds because she had maligned and insulted a Southern lady.”

Carr admittedly was uncommonly explicit about conflating Confederate memorialization with white supremacy, but Southern memorials inherently celebrated the slave South and white power along with the heroism of Confederate soldiers.

We topple old buildings, move or rename streets, and engage in creative destruction all the time — which is inevitable when the needs of the people living contemporary landscapes change. The somewhat comical events in Reidsville (in which the United Daughters of the Confederacy concluded it would be for the best if fewer drivers crashed into their statue) provide just one example of a decision to move a memorial for a practical reason.

Elsewhere, communities have had other reasons to act. Wilson, North Carolina, for example, has been home since 1926 to a memorial that commemorated the Revolution and the Confederacy: It originally featured a massive central column depicting the Stars and Stripes and the flag of the Confederate States of America, flanked by two water fountains — one for whites, one for blacks. It apparently outlasted its welcome sometime during the 1960s. Without fanfare, the fountain was moved from the court house to an inconspicuous park, and the fountains were replaced by small granite caps. Today you would be unlikely to recognize it as a one-time segregated water fountain.

So how should we move forward to dismantle the Confederate commemorative landscape? We should begin by acknowledging that the American South is now a pluralist society for the first time in its history. Whereas the current commemorative landscape of the South is a product of white privilege and power, the future landscape should be crafted after inclusive public debate and through democratic procedures. New Orleans and Baltimore, which conducted public conversations about the removal of monuments, can serve as models for other communities. New Orleans Mayor Mitch Landrieu has provided an exceptionally articulate justification for the removal of Confederate memorials.

A crucial step in many Southern states will be to repeal laws constraining the removal or alteration of historic monuments, such as North Carolina’s two-year-old Historic Artifact Management and Patriotism Act. Let there be no doubt about the intent of this or similar “heritage preservation” laws: They “protect” and perpetuate the racist commemorative landscape that currently exists. Why shouldn’t the citizens of Durham have had the choice to preserve, move, or remove the Confederate monument there? Local choice may allow some communities to keep “their” Confederate monuments. So be it. Let them defend their decision if they do so.

We are also sure to hear calls to add monuments (honoring African Americans, for example) as an alternative to removing those we find offensive, and thereby “erasing” history. But removing — or moving — Confederate monuments is not historical erasure. The same logic could have been used to justify maintaining, after 1964, signs that identified “Negro water fountains,” “Colored waiting room,” and the other markers of Southern segregation.

In an ideal world with unlimited resources, a proposal to add monuments might make sense. But given the vast number of monuments to the Confederacy across the United States it would take decades, and millions of dollars, to add enough statuary to create a more inclusive commemorative landscape. And is there any reason to believe that state legislators are going to appropriate sufficient money for that purpose? Perhaps the defenders of Confederate monuments will demonstrate their good faith by pressing for funding for new monuments to Southerners, white and black, who fought on behalf of the Union or otherwise opposed the Confederacy. Until then, I will view their devotion to heritage preservation with skepticism.

This is hardly the first time that a society has confronted the issue of dealing with art harnessed to objectionable causes. Art museums are filled with medieval and early modern Western art that is offensive to many of our contemporary values — depicting rape, the slaughter of Muslims, or demeaning images of non-Europeans. Like those works of art, those Confederate monuments that have aesthetic significance can and should be preserved in museums where they can be properly interpreted by curators and docents. In such settings, they will serve as historical artifacts rather than civic monuments.

But many Confederate monuments were essentially “mail order” sculptures mass produced by Northern and Southern foundries during the late 19th and early 20th centuries. Whatever value they have as historical artifacts, they were not the work of some latter-day Michelangelo.

Before any Confederate monuments are removed, they should be carefully photographed and measured so that the historical record of the monuments in situ can be preserved and made available for historians and art historians in the future. Then they can be transferred to the archives, museums — or the trash heap of history.

W. Fitzhugh Brundage is the William B. Umstead Professor of History at the University of North Carolina at Chapel Hill. He is the scholarly adviser to the Commemorative Landscapes of North Carolina project.

The Big Idea is Vox’s home for smart discussion of the most important issues and ideas in politics, science, and culture — typically by outside contributors. If you have an idea for a piece, pitch us at thebigidea@vox.com.