As a son of the South, I’m glad to see folks finally getting their act together on the hundreds of Confederate monuments spread around the region — folks like the Republican governor of Tennessee, Bill Haslam, who recently endorsed removing a bust of Nathan Bedford Forrest from the state Capitol. That’s low-hanging fruit; Forrest was a Confederate monster who massacred black soldiers and later helped found the Klan. Still, it’s a start.

But I also know that these monuments, taken as a whole, are themselves low-hanging fruit. I grew up in Nashville, where, within 10 minutes on my bike, I could be riding along Jefferson Davis Drive, Confederate Drive, General Forrest Court, Robert E. Lee Court and, confusingly, two separate Robert E. Lee Drives. My Boy Scout troop went to the national jamboree at Virginia’s Fort A. P. Hill, named for a Confederate general. A few classmates went to college at Washington and Lee. You get the point — removing the legacy of the Confederacy is harder than toppling a few statues.

The difficulty doesn’t lie just in changing people’s addresses or diplomas. I grew up in a decently liberal family and was taught all the right notions about tolerance and civil rights and the awful history of slavery. But somehow, as a kid, riding my bike along Robert E. Lee Drive never really bothered me. I knew full well what Lee stood for; had I stopped for a second, I might have imagined how it would feel to be a black person riding along that same street. But that’s the point: I didn’t, because it all just seemed so natural, so all encompassing. Not ideal, but unavoidable.

So when people ask why, in 2017, most white people in the South don’t do more to confront the memory of the Confederacy, that’s why. Yes, some small number are white supremacists. A larger number, though still a minority, cling to the idea that the memory of the Confederacy is about “heritage, not hate.” For most, I’m convinced, it’s like a slight stink in the air. Unpleasant, perhaps, but everywhere, and so it’s something you don’t think or do much about, and don’t understand the fuss when someone does.