Last weekend’s white nationalist rally in Charlottesville, Virginia, began as a protest against the city’s plan to remove a statue memorializing Confederate general Robert E. Lee. Efforts to remove such monuments are nothing new, but they have intensified in recent years and look likely to gain further steam in the wake of the deadly events in Charlottesville. On Tuesday night, the city of Baltimore began removing its Confederate monuments, including statues of Lee and Stonewall Jackson. Gainesville, Florida, removed a monument to Confederate soldiers this week, and the mayor of Lexington, Kentucky, said Saturday that he was accelerating plans to remove two Confederate statues there. And in some cities, such as Durham, North Carolina, protesters are taking matters into their own hands by toppling these statues themselves.

They have their work cut out for them. There are currently more than 700 monuments to the Confederacy in public places, located predominantly in the South. Only a tiny fraction of them have been removed so far. And as this weekend’s protests showed, efforts to take the monuments down — or even to relocate them to less prominent locations — often encounter vocal opposition. President Trump himself on Tuesday seemed to question whether Confederate monuments should be removed. “So this week, it’s Robert E. Lee, I noticed that Stonewall Jackson’s coming down,” Trump said at a press conference at Trump Tower. “I wonder, is it George Washington next week? And is it Thomas Jefferson the week after. You know, you really do have to ask yourself, where does it stop?”

The movement to remove Confederate monuments gained traction after the deadly 2015 shooting in a black church in Charleston, South Carolina. After the attack, photos surfaced of the shooter, Dylann Roof, posing with a Confederate battle flag. Within weeks of the incident, the South Carolina House had voted to remove a Confederate flag at the statehouse, a move that was also met with protest. Since the shooting, statues have been removed or relocated in Kentucky, Louisiana, Maryland and other states; multiple monuments, memorials and flags have also been removed or are slated for removal. Numerous public spaces with names honoring the Confederacy, including schools, parks and streets, have also been renamed.

“Charleston was a fulcrum moment,” said Alfred Brophy, a professor at the University of Alabama School of Law. “The steam with which monuments are coming down has accelerated greatly. There’s more public discussion. Where before it was done more quietly, now there’s a sort of celebration.”

In the wake of the shooting in Charleston, the Southern Poverty Law Center began collecting data on public displays of the Confederacy throughout the United States. Using federal, state and other sources, they found more than 1,500 places or things commemorating the Confederacy, including more than a hundred schools and more than 700 monuments. The SPLC’s list of symbols also includes street and county names, as well as parks, military bases and a broad range of other public works or spaces. The vast majority are located in states that once made up the Confederacy, though they extend north and west as well.

Most of those monuments and other symbols date back not to the immediate aftermath of the Civil War but to the early 20th century, when many Southern states were imposing Jim Crow laws. Another wave of monument-building came during the civil rights movement of the 1960s, and it hasn’t fully ceased — the SPLC database includes a monument to Confederate soldiers in Tennessee that was erected in 2012.

Despite the renewed attention to the issue since the Charleston attack, only a relative handful of these monuments have been removed. The SPLC had identified just six cases in which statues or monuments have been taken down before last weekend’s events, including the removal of four statues in New Orleans that prompted a much-discussed speech from the city’s mayor. That number has grown by at least six more this week alone, reflecting growing momentum after the events in Charlottesville. And the SPLC has identified dozens more symbols of various kinds that have been or might be removed.

Those in favor of removing public Confederate symbols say those symbols represent oppression and slavery, and that these parts of history “belong in a museum, not on a pedestal.” But in the past, states have fought in support of the preservation of Confederate monuments. Conservatives who protest for the preservation of these monuments assert that they are part of Southern heritage and reflect a part of history — good or bad. Several states, including South Carolina, Georgia and North Carolina, have heritage protection acts that restrict the removal of any monument located on public property.

The hate and violence on display in Charlottesville, however, could change the debate. Brophy, the law professor, has previously argued against the wholesale removal of Confederate monuments, which he has said represent a reminder of the nation’s troubled history. But he said the weekend’s white supremacist demonstration shows that the monuments are also serving as a present-day rallying cry for violence. After Charlottesville, Brophy said it’s unlikely politicians will stand in the way of the monuments’ removal, and in some cases, as in Durham, protesters will ignore those laws altogether. “This is the sign that monuments matter,” he said.

Others, however, worry that the movement to take down the monuments has potential to backfire by providing a platform for white nationalist protesters. And even if the statues do come down, some civil rights activists and others question how much difference it would make. Susannah Ryan, a University of North Carolina doctoral student who has studied conflict and reconciliation, said the fight over statues and monuments gives politicians a way to call for equality without addressing harder, more fundamental issues such as voting rights.

“As we see some of these monuments being torn down, another question we have to face is to what extent is that just a veneer,” Ryan said.