CHARLOTTESVILLE, Va. -- About 50 Ku Klux Klan members protested Saturday in Charlottesville, Virginia, after the city decided to sell a park statue of Confederate Army General Robert E. Lee but they were dwarfed by an estimated 1,000 counter-protesters.

The Loyal White Knights of the Ku Klux Klan organized to protest the city's decision to take down the statute and rename the park around it, and orginally expected 80 to 100 Klan attendees. While there was no reason given for the smaller turnout, the Anti-Defamation League says the group has been diminished by infighting and high turnover, CBS News' Paula Reid reports.

A total of 23 people were arrested Saturday, according to the city. Some counter-protesters were reluctant to leave the space when the rally's designated time slot ended which prompted law enforcement to declare an unlawful assembly. Both the Virginia State Police and the Charlottesville Police Department were on the scene with more than 100 officers.

Get Breaking News Delivered to Your Inbox

Some of the Klan members were armed with guns and some wore Klan robes at Saturday's rally.

Counter-protesters shout at members of the Ku Klux Klan, who are rallying in opposition to city proposals to remove or make changes to Confederate monuments, in Charlottesville, Virginia, U.S. July 8, 2017. Reuters / Jonathan Ernst

In April, the Charlottesville City Council voted 3-2 to sell the statue of General Lee; however, the statue is not currently for sale because in May, Charlottesville judge Richard Moore ruled that in the public's interest, the Lee statue cannot be moved for a period of six months. A court decision is expected sometime in November.

On the eve of the planned protest, the statue was vandalized with red paint.

The Klan requested a logistical permit and the city staff worked with the Klan to find an appropriate place.

"People are guaranteed the right to assemble, and the park is a public place," said Miriam Dickler, communications director for the city. "Folks are allowed to come and say their piece."

Riot police protect members of the Ku Klux Klan from counter-protesters as they arrive to rally in opposition to city proposals to remove or make changes to Confederate monuments in Charlottesville, Virginia, U.S. on July 8, 2017. Reuters / Jonathan Ernst

Throughout the city of Charlottesville, the group "Unity C-Ville" held counter-programming efforts to encourage locals to not attend the rally. One Unity C-Ville organizer, local brewery owner Hunter Smith, said that many in the city had a "visceral response" when they heard about the KKK rally, so community leaders wanted to "bring people together in a peaceful way," away from the park demonstration.

The Charlottesville statue is one of many recently in the news as cities try to grapple with the legacy of such monuments. In St. Louis in June, a Confederate memorial was removed from Forest Park after an agreement was reached between the city and the Civil War Museum. In New Orleans, four statues were removed after two years of court battles.

One protester wearing a Confederate flag hat told Reid that taking down these statues is an effort to re-write history. "You can't erase history," he said.

Klan protesters at a rally in Charlottesville, Va., on July 8, 2017. Bo Erickson

According to a report from the Anti-Defamation League, the organized Ku Klux Klan movement in the United States currently "consists of just over 40 active Klan groups, a slight increase from early 2016. More than half of the current Klans have formed in the last three years.

Charlottesville, a small city of about 43,000 people, is home to former President Thomas Jefferson's Monticello residence and the university he founded, the University of Virginia.

"The Klan and the NAACP have a deep history," said Janette Martin, president of the local NAACP chapter. "I never thought we would go down this path again in 2017."

CBS News' Paula Reid and Kelsey Micklas contributed to this report.