When the jazz musician Wynton Marsalis first broached the idea of removing the Confederate monuments in New Orleans to Mayor Mitch Landrieu in 2014, the mayor was skeptical.

“I really rejected it, to tell you the truth,” Landrieu told TIME in an interview Wednesday. But he says it soon became “painfully obvious” that several of New Orleans public monuments’ commemorating Reconstruction-era white uprisings and defenders of slavery had to go. “The truth was that these monuments were put up purposefully to send a message and to be a sanitized version of the Confederacy,” Landrieu says.

Two-and-a-half years later, following a city council vote and a series of ongoing legal challenges, the monuments in New Orleans have started falling—and the removal has awakened the ghosts that linger in the South, where some see tributes to the Confederacy as symbols of heritage and others as reminders of hate.

The latest came early Thursday morning, when a crew of workers protected by police began removing a 1911 statue of Confederate President Jefferson Davis. Scattered protesters yelled “cowards” at the workers while supporters cheered, according to the Times-Picayune. It followed the late April removal of Liberty Monument, a granite obelisk memorializing the 1874 uprising of the Crescent City White League, which attempted to overthrow the state’s Reconstructionist government. The crew tasked with the work hid their faces with cloth and covered the logos on their trucks with cardboard, while law enforcement snipers kept watch from nearby roofs. “Our history is not a game of Jenga!” one man shouted as the granite obelisk was pieced apart.

Read more: New Orleans Removes First of 4 Confederate Statues: ‘This is Not About Politics’

Two other memorials—statues of Gen. Robert E. Lee and Gen. P.G.T. Beauregard—will likely come down soon, and the city will be taking similar precautions to protect the workers and keep protests from turning violent.

“There’s no other place in the world that pays tribute and puts on a pedestal people that tried to destroy our country,” Landrieu says. “It would be the equivalent of putting King George where the Washington Monument is or putting Robert E. Lee where Abraham Lincoln is.”

The removal effort has received strong backing from both the mayor and the city council, as well as advocacy groups such as Take ‘Em Down NOLA, which is pushing for the city to go beyond the four monuments slated to come down. It wants to eliminate 23 statues and hundreds of names of slaveholders and Confederate soldiers from public schools and streets. “We should not be venerating Confederates or bestowing an honor to them,” says resident Malcolm Suber, a leader of the group.

The city council voted in favor of Landrieu’s removal proposal, 6-1, in December 2015, six months after the deadly Charleston, S.C. church massacre by white supremacist Dylann Roof. While the shooting helped galvanize public sentiment against civic displays of the Confederacy, Suber also attributes the vote to “the political maturity of the black electorate.”

New Orleans today is 60% black, and it, along with many other southern cities, has seen its ranks swelled by outsiders who may not necessarily share the region’s deep-rooted sense of history. In the last two decades, millions of people have moved to the South from other parts of the country, helping to shift the region’s politics. “The constituency for these monuments has diluted significantly,” says David Goldfield, a history professor at the University of North Carolina at Charlotte who studies Confederate symbolism. “Those who want the monuments in their current position really feel that their status has been eroded.”

Read more: New Orleans to Remove Confederate Statues

The New Orleans monuments slated for removal were all erected between 1884 and 1915, a period when lynchings spiked and some of the most vicious Jim Crow laws were enacted. The Liberty Monument, for example, at one point had a plaque recognizing “white supremacy in the South.”

To the monument’s defenders, however, they represent an important link to the city’s history and taking them down is equivalent to ripping out pages from a textbook. Pierre McGraw, president of the Monumental Task Committee, which has been working to keep the monuments up, says they are not symbols of white supremacy and cites a poll by Louisiana State University showing that more than 70% of the state’s residents oppose their removal. His group has offered a three-point plan for keeping them in place, which includes interpretative plaques to help put the monuments in context.

“When we trash our historic cultural artifacts, that doesn’t serve a common good,” McGraw says. “It tells me that New Orleans has lost her soul.”

Landrieu says he is holding firm, even after a march organized by monument opponents led to clashes with supporters outside the Lee statue on May 7. But he acknowledges that the political climate is different from when the city council signed off on his proposal.

“There was a fairly dignified bringing down of the flag in South Carolina,” he says. “That didn’t happen here,” referring to the sometimes-violent clashes outside Lee’s statue.

The mayor wouldn’t comment on whether the monuments will eventually be placed into a museum, only saying that the remaining two would come down “sooner rather than later.”

“We’re actually correcting history,” he says. “Those monuments have never reflected the totality of who we are.”

Get The Brief. Sign up to receive the top stories you need to know right now. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Contact us at letters@time.com.