You don’t have to be white to be accused of being a white supremacist, as some of those caught up in the rush to denounce racists and fascists in the wake of Charlottesville can attest.

Take Shiva Ayyadurai and Daniel Alejandro Medina. Neither is white, but both were featured speakers at Saturday’s small free speech rally in Boston, which drew an estimated 15,000 protesters determined to counter what had been widely billed by foes as a white supremacist gathering.

That’s not what video clips show. At one point Mr. Ayyadurai, a candidate for the 2018 Republican Senate nomination in Massachusetts, declared that “black lives matter” as his supporters held up signs with the messages “Black Lives Do Matter” and “No to GMO’s: Stop Monsanto.”

What’s more, the event opened with a moment of silence for Heather Heyer, the woman killed Aug. 12 in Charlottesville, Virginia, by an alleged neo-Nazi charged with driving his car into a crowd.

“They took advantage of Charlottesville to start screaming that this was a neo-Nazi, white supremacist rally when it’s so ridiculous,” said Mr. Ayyadurai, an entrepreneur who holds four degrees from MIT. “I’m a brown-skinned Indian immigrant, low-caste untouchable — I don’t know how many labels you want to put on me — who’s fought all his life.”

That such an assembly would be described even afterward as a kind of racist pep rally — Boston Mayor Marty Walsh thanked those who came to “fight back on the white supremacists that were coming to our city” — illustrates the challenge for conservatives and free speech activists as progressives and Democrats seize on the white supremacy narrative.

If anyone can empathize, it’s Joey Gibson. The founder of Patriot Prayer, he has been condemned by no less than House Minority Leader Nancy Pelosi for organizing a Freedom Rally slated for Saturday at San Francisco’s Crissy Field, located on federal land.

She described it as a “white supremacist rally” in an Aug. 15 statement calling on the National Park Service to reconsider the permit, even though Mr. Gibson is biracial — one of his parents is of Japanese descent — and he says he has denounced racism “a million times.”

“Describing me as [part of] an alt-right group is just an attempt for them to smear me. That’s why they’ve picked it up a notch, and now they’re saying that I’m a white supremacist group,” Mr. Gibson said. “It’s really frustrating.”

Mayor Ed Lee and other city officials have urged San Franciscans to stay away from the rally, countering with a block party at City Hall scheduled for the same time Saturday as the Patriot Prayer event.

“I ask that people avoid going to Crissy Field and engaging with members of Patriot Prayer, because that is precisely what they wish us to do, and I don’t want to dignify their message of hate and their mission of division in our city of love [and] compassion,” Mr. Lee said at a Wednesday press conference.

Mr. Gibson, who founded Patriot Prayer earlier this year to offset anti-Trump rallies in the Pacific Northwest, pointed out that his event features a racially diverse list of scheduled speakers.

“We have one white male speaking. The rest are black or Hispanic, and we have a transsexual,” said Mr. Gibson, who lives in Vancouver, Washington. “No way is it a white supremacist rally.”

His critics argue that such gatherings, even if not sponsored by actual white nationalists, nonetheless attract neo-Nazis and others with racist views.

In an Aug. 7 report on a Portland rally, the liberal Southern Poverty Law Center noted that organizers “promised the critics who talked with them that racist elements had been denounced and uninvited from the rally.”

“Yet in addition to the Proud Boys, a number of Portland-area activists from Identity Evropa — the openly white-supremacist alt-right student group — including local leader Jake Van Ott were visibly present in the crowd,” said the SPLC.

Mr. Gibson acknowledged that those affiliated with Identity Evropa have shown up at his rallies, which typically number fewer than 100, but said that he has made it clear they aren’t welcome and kicked them out whenever possible.

“It’s a constant problem because we get these random people that are trying to provoke and they’re trying to agitate,” Mr. Gibson said. “There are people who hate how much I’m in the middle. They hate how much I’m trying to reach out. There are so many people who leave me because I’m too nice and I’m communicating with liberals too much or antifa and communists.”

The SPLC, which does not list Patriot Prayer on its “hate map,” later commended him for inviting protesters onstage to speak at an Aug. 13 rally in Seattle.

Mr. Gibson said that if anyone is endangering the public, it’s elected officials like Mrs. Pelosi who have stirred unrest over the event, virtually guaranteeing that the rally will be mobbed by far-left groups like the antifa.

“It’s really sad because not only is it immoral, but they’re putting people’s lives in danger,” Mr. Gibson said. “Nancy Pelosi, she’s doing that. She’s trying to further her own agenda, and she’s putting her own citizens in danger, directly in danger, [over something] which she knows to be a lie.”

The Boston rally was mocked on social media for its low attendance — only about 50 people made it onto the Parkman Bandstand in Boston Common — but participants blamed that on the city.

Mr. Ayyadurai said the permit limited attendance on the bandstand to 100, but that the press and hundreds of supporters were barred by police from watching the heavily barricaded event on the surrounding lawn.

After leaving the rally, Mr. Ayyadurai spoke to reporters outside the gazebo, at one point asking counterprotesters booing and interrupting him to be quiet.

“I think people like myself as a person of color are tired of being called Nazis and white supremacists,” said Mr. Ayyadurai, as shown on a YouTube video. “I want to stand up for free speech.”

• This article is based in part on wires reports.

Sign up for Daily Newsletters Manage Newsletters

Copyright © 2020 The Washington Times, LLC. Click here for reprint permission.