Update: The “free speech” rally planned for Saturday has been cancelled. Read more here.

Residents of San Francisco are bracing for thousands to descend on the city’s streets this weekend — and hoping that there will not be violence. Weeks after a deadly confrontation between white nationalists and counter-protestors in Charlottesville, Va., right-wing organizers are hosting a “free speech” event near the Golden Gate Bridge on Saturday, and left-wing organizers have planned more than a dozen events around the city in response.

Though the organizers of the “free speech” event have disavowed white nationalism and say their intention is to gather peacefully, many fear that neo-Nazis will show up, a specter that has sent San Francisco’s progressive activists into overdrive. “People are very alarmed by what’s going on in the country,” says longtime activist Cleve Jones, a well-known figure in the gay rights movement. “I have never seen the level of activity that I’ve seen over the last year.”

On Wednesday, the National Park Service issued a finalized permit for the “free speech” rally to a right wing group known as Patriot Prayer, allowing the event planned for Crissy Field to go forward. The expanse that runs along the city’s northern shore is federal land, and the park service said that after consulting with their law enforcement arm, as well as the San Francisco Police Department, the decision was made to issue a permit “to promote the peaceful, non-violent expression of views.”

The permit also came with conditions: not only will firearms be banned (normally concealed carry laws allow them in the area) but ralliers will not be allowed to bring anything that might be used as a weapon — pepper spray, drones, shields, bats, selfie sticks, as well as other items.

City officials do not have estimates for how many people might show up. But grassroots organizers believe there could be as many as 50,000 people in attendance this weekend, the vast majority taking part in counter-demonstrations that range from a musical celebration in Civic Center to a “mobile dance party” to the formation of a 100-foot “human heart” on Ocean Beach. Another right-wing rally planned for Sunday in Berkeley is expected to draw additional crowds from both sides.

Joey Gibson is the man behind Patriot Prayer (which he says is an “ideology” more than a group). His guess is also that the people at his rally will be vastly outnumbered by counter-protestors, and it’s an educated one: The 33-year-old from Portland, Ore., has organized about a dozen similar events in the last several months.

Though they began as pro-Trump rallies, Gibson says he’s left issues of “who you vote for” behind and is now focused on the belief that the First Amendment is being threatened by people who are intolerant of any viewpoints that don’t match their own, on the left and the right. “People have a right to say whatever they want,” says Gibson. “Hate speech is free speech … Everyone has a right to be hateful. Unfortunately it’s not good for our society. But there is no debate, the law is the law.”

While he has publicly disavowed white supremacy and racism, Gibson also casts anti-fascist organizers known as antifa as one of the enemies of free speech — a belief that many white nationalists share. He acknowledges that individuals with extreme views have been attracted to his events in the past and that part of the intention of hosting an event in San Francisco is to show how “extremely intolerant” such liberal cities can be.

“There’s a lot people who disagree with the culture but they’re too afraid to stand up to it because they get called names, they get called Nazis, racists,” he says. There are people in cities like San Francisco who have “a meltdown when I’m sending a good message … they’re going to get triggered and they freak out. That stuff gets on film and we send that film out so the country can see, they can see the insanity.” Such statements give credence to the notion that these rallies are being held to provoke the left, even if Gibson has spoken often about love and peace.

While officials are working to keep opposing groups far apart from each other, hoping the weekend ends without serious injuries as recent gatherings in Boston did, many expect extremists on the right and left to clash. Local lawmaker Mark Farrell, a driving force behind a “Peace, Love and Understanding” rally being hosted in the city’s Civic Center, says his intention is to draw counter-protestors far from Crissy Field so that people can “celebrate our spirit in San Francisco in a way that does not give them any oxygen.”

Like other city officials, Farrell has cast the Patriot Prayer event as a “Nazi rally” that will be catnip for white supremacists. When asked what he thinks of Gibson’s statements about opposing such groups and promising to turn them away, he says, “I don’t buy what they are selling for a minute.”

His office oversees the area of the city that abuts Crissy Field and has been dealing with an influx of phone calls and emails from residents who are “angry and frightened,” he says. “We want to do everything possible to encourage people not to show up at the event.”

Local organizers have been grappling with how to balance safety and the desire that residents in San Francisco have to make a statement about their beliefs. On a recent conference call, organizers from several groups swapped tips about how to maintain non-violence and traded ideas about what form protest should take.

Some of those ideas were high-minded, like setting up a website where people can, from the comfort of their homes, donate to groups that have missions antithetical to neo-Nazis. Others were lower brow, like taking dogs to poop en masse in Crissy Field before the rally was supposed to start. A suggestion to throw glitter on the rally’s speakers was scrapped when someone pointed out that the glitter might hurt the “sensitive environmental habitat” in the area.

One of the people on that call was Megan Rohrer, a Lutheran pastor and volunteer chaplain for the SFPD. Rohrer, who uses the pronoun they, says some people also imagined unfurling a giant rainbow flag from the Golden Gate Bridge, which will be the backdrop for the “free speech” event.

Rohrer acknowledges that “there might be people who want to go there to have a violent encounter.” Many expect anti-fascist groups to meet the ralliers head on. But Rohrer also hopes people will attend a march from the city’s Castro district to Civic Center instead. “The more we can feed people sandwiches rather than an adrenaline,” they say, “the better our chances for keeping peace.”

Get The Brief. Sign up to receive the top stories you need to know right now. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Write to Katy Steinmetz at katy.steinmetz@time.com.