Let’s look at real history. Americans tend to assume that Southern segregation was a “natural” legacy of antebellum slavery. The truth is far more complicated. After the Civil War, the South went through a period of transition—not simply during “Reconstruction” (which ended in about 1877) but for the two decades that followed. There was no overall “system” of separation; gross racism and discrimination existed alongside tentative inter-racial cooperation and political coalition-building. Until the first decade of the twentieth century, black Southerners continued to vote, to serve on juries, and to hold state and local office. The last “first-generation” black member of the U.S. House left office in 1901.

Only with the rise of the U.S. as an imperial power—forcibly dominating people of color from San Juan to Manila—did the idea of legal white supremacy become acceptable to a majority of whites in North or South. Thus began the era of segregation—a system that subordinated black Southerners economically, disfranchised them politically, and isolated them in public and private space. What’s called the “nadir” of race relations was the early 20th Century, not the 1870s and 80s.

The year 1890 saw the first segregation-era Southern state constitution, in South Carolina, strip blacks of the right to vote. That same year, the giant Lee statue went up in Richmond. Virginia itself disfranchised black voters in 1902. The monuments to Jefferson Davis and Jeb Stuart went up in 1907; the horseback statue of Jackson was unveiled in 1919. All across the South during these years, these statues went up to mark the triumph of the once-outlandish idea of segregation.

Segregation had an official myth: The white South would have freed its slaves voluntarily if not for Northern meddling. The North destroyed the South because it coveted its natural resources and its cheap labor. After the War, corrupt “carpetbaggers” and vile Southern white “scalawags” seized power with Northern bayonets, upheld by ignorant, illiterate blacks. Heroic white conservatives finally did away with the “corrupt Negro vote,” restored to power the South’s natural leaders, and returned black Southerners to their proper subordinate place.

Blacks played no part in any of Southern history. They had no past, and no future, in white America.

As New Orleans Mayor Mitch Landrieu said in his recent speech, “These statues are not just stone and metal. They are not just innocent remembrances of a benign history. These monuments purposefully celebrate a fictional, sanitized Confederacy; ignoring the death, ignoring the enslavement, and the terror that it actually stood for.”

So, what is to be done? Because I know Richmond, I use it as an example, more extreme than many but not different in kind from most cities across the South. Richmond’s new mayor, Levar M. Stoney, recently said, “I want to be a city that is tolerant, inclusive and embraces its diversity, and those statues without context do not do that.” Stoney has called for “a robust community conversation about it that will involve stakeholders and community groups and residents.”