(CNN) Protests are rattling genteel Charlottesville, Virginia, hometown of Founding Father Thomas Jefferson, over the city's plans to remove a statue of Confederate Gen. Robert E. Lee from a downtown park.

On Saturday night, a group carrying torches and led by prominent white nationalist Richard Spencer held a demonstration around the statue to protest its removal. The gathering was swiftly condemned by city leaders, who said it evoked images of the Ku Klux Klan.

"This event involving torches at night in Lee Park was either profoundly ignorant or was designed to instill fear in our minority populations in a way that hearkens back to the days of the KKK," Charlottesville Mayor Mike Signer said in a statement . "I want everyone to know this: We reject this intimidation ... such intolerance is not welcome here."

That was followed by a candlelight counterprotest Sunday night, where speakers preached tolerance and protesters hung a banner on the statue's base that read, "Black Lives Matter" and "F**k White Supremacy."

Police made three arrests after several right-wing protesters showed up and a scuffle broke out. A police officer was injured when an object thrown from the crowd struck him in the head

The tensions have made this historic city, home of Monticello and the University of Virginia, the latest Southern battleground over the contested removal of Confederate monument s -- symbols of the Civil War that represent heritage to some, hate to others.

Torch-carrying demonstrators gather in Charlottesville on Saturday to protest the statue's removal.

In Charlottesville, it all started when the city council voted narrowly in February to remove the bronze statue, which was installed almost a century ago and depicts Lee in uniform astride a horse. The city also voted to rename Lee Park, where the statue stands, and another nearby park named for Confederate Gen. Thomas "Stonewall" Jackson.

A group of residents have sued the city to keep the Lee statue in place. A judge has agreed to block its removal for at least six months while the litigation proceeds.

People who support removal of the monuments hold candles during a counterprotest on Sunday.

Meanwhile, protesters are converging on the city's parks. Alt-right darling Spencer, a University of Virginia graduate, led a rally Saturday afternoon in Jackson Park, where he told several dozen supporters, "We will never back down from the cowardly attacks on our people and our heritage. What brings us together is that we are white. We are a people. We will not be replaced!"

His words, and the evening torch rally, were quickly denounced by residents and officials -- both Democrat and Republican.

"You're not going to drown us out, you're not going to make us listen to you. There is no such thing as a supremacy. Slavery is dead and we just need everyone to realize and recognize that," said protester Don Gathers at Sunday's candlelight rally, according to CNN affiliate WVIR

Two local groups who support preserving the Lee statue also condemned Saturday's protests.