Virginia's governor declared an emergency, and police worked to disperse hundreds of protesters in Charlottesville after clashes broke out ahead of Saturday's "Unite the Right" rally of white nationalists and other right-wing groups.

Fistfights and screaming matches erupted shortly before rally's scheduled noon ET start. The skirmishes unfolded following a scuffle Friday night between torch-bearing demonstrators and counter-protesters at the nearby University of Virginia.

Saturday's rally was the latest event drawing white nationalists and right-wing activists from across the country to this Democratic-voting college town -- a development precipitated by the city's decision to remove symbols of its Confederate past.

Here are the latest developments:

• President Donald Trump tweeted: "We ALL must be united & condemn all that hate stands for. There is no place for this kind of violence in America. Lets come together as one!"

• Police began to break up crowds shortly before noon after city officials declared the gathering an "unlawful assembly." Police officers spoke on bullhorns, directing people to leave.

• The declaration was made after fistfights and screaming matches erupted in several locations late Saturday morning.

• Some protesters fired pepper spray at other demonstrators, state police said.

• Virginia Gov. Terry McAuliffe declared a state of emergency "to aid state response to violence," according to a post on his Twitter account.

• An unspecified number of protesters have been arrested in Charlottesville, state police said.

Police in riot gear stood shoulder to shoulder behind shields early Saturday afternoon, at times advancing toward crowds, CNN video shows. Members of the Virginia National Guard also were there.

It wasn't immediately clear what led to the fights, though tensions and rhetoric were running hot. At one point, a few dozen white men wearing helmets and holding makeshift shields chanted, "Blood and soil!" Later, another group chanted slogans like, "Nazi scum off our streets!"

People punched and kicked each other during various scuffles, which often were broken up from within crowds, without police intervention, CNN video shows.

Earlier, a group of clergy and other counter-demonstrators, including activist and Harvard professor Cornel West, held hands, prayed and sang, "This Little Light of Mine."

Police presence was heavy, with more than 1,000 officers expected to be deployed, city officials said. Police anticipated the rally would attract as many as 2,000 to 6,000 people, and the Southern Poverty Law Center said it could be the "largest hate-gathering of its kind in decades in the United States."

White nationalists wield torches

Charlottesville, once home to Thomas Jefferson, is known as a progressive city of about 47,000 people. Eighty percent of its voters choose Hillary Clinton during last year's election.

But far-right activists and Ku Klux Klan members have come here in recent months, outraged by the city's intention to remove traces of its links to the Confederacy -- including plans to remove a statue of Confederate Gen. Robert E. Lee.

The move follows efforts by communities across the South to remove Confederate iconography from public property since the 2015 rampage killings of nine black churchgoers in Charleston by a self-described white supremacist.

Ahead of Saturday's planned rally, tensions roiled Friday night as white nationalists -- some holding what appeared to be backyard tiki-style torches -- marched onto the University of Virginia's campus.

Chanting, "Blood and soil" and "You will not replace us," the group rallied around a statue of Thomas Jefferson before they clashed with counter-protesters, CNN affiliate WWBT reported. The group left the university's grounds when police arrived and declared the gathering an unlawful assembly.

City and UVA officials condemned Friday's march.

"In my 47 years of association with @UVA, this was the most nauseating thing I've ever seen. We need an exorcism on the Lawn," Larry Sabato, director of the university's Center for Politics tweeted.

Charlottesville Mayor Mike Signer released a statement referring to Friday's rally as a "cowardly parade of hatred, bigotry, racism, and intolerance march down the lawns of the architect of our Bill of Rights."

"Everyone has a right under the First Amendment to express their opinion peaceably, so here's mine: not only as the Mayor of Charlottesville, but as a UVA faculty member and alumnus, I am beyond disgusted by this unsanctioned and despicable display of visual intimidation on a college campus," he added.

Friday's march took place shortly after a federal judge granted a temporary injunction allowing right-wing activists to hold Saturday's rally.

City officials had tried to "modify" the rally's permit to move the demonstration from the park with the Lee statue more than a mile away to McIntire Park, citing safety concerns.

'We're going to start standing up for our history'

In February, the city council voted to remove the Lee statue, but that is on hold pending litigation. The council also voted to rename two city parks that had been named for Confederate generals; one of those, Emancipation Park, was due to be the site of Saturday's rally.

Saturday's event had residents on edge, and more than 40 local business owners near the park have asked the city to protect them.

"I have a lot of fears. I think most of us are just anxious, we don't want there to be violence," business owner Michael Rodi said of the rally.

"We don't want to see a bloodbath, we don't want to see looting, we don't want to see mass arrests, we don't want to see the police having to turn on citizens," he added.

Jason Kessler, who organized Saturday's "Unite the Right" rally, said he doesn't consider himself to be a white nationalist. But, he said, "we're going to start standing up for our history."

"The statue itself is symbolic of a lot of larger issues. The primary three issues are preserving history against this censorship and revisionism -- this political correctness," he told CNN Friday.

"The second issue is being allowed to advocate for your interests as a white person, just like other groups are allowed to advocate for their interests politically. And finally this is about free speech. We are simply trying to express ourselves and do a demonstration, and the local government has tried to shut us down."