Clergy in Boca Raton, Florida are speaking out against a local elections official’s decision to remove a polling site from a mosque, saying Muslim houses of worship shouldn’t be treated any differently than churches or synagogues.

“In my estimation, the worst thing that can happen is that we play by different sets of rules,” Rabbi Anthony Fratello told TPM in a phone interview this week. “If you’re going to say that people don’t want to go into someone else’s house of worship to vote, polling centers shouldn’t be in houses of worship.”

That comment by Fratello, whose Temple Shaarei Shalom in Boynton Beach has served as a polling site since 2004, was echoed by other faith leaders who have appealed to Palm Beach County Elections Supervisor Susan Bucher to reinstate the Islamic Center of Boca Raton as a polling place.

Bucher initially invited the center to receive voters in April before rescinding that invitation in July over complaints from “uncomfortable” residents. The Islamic center’s president, Bassem Alhalabi, told TPM he feels Bucher’s decision to remove the site amounted to “surrendering to bigotry.”

Leaders at Boca Raton’s Temple Beth El urged Bucher to change her mind in a letter last week, cautioning her against giving into “the most negative stereotypes and presumptions about Islam.” An interfaith coalition plans to send another letter to Bucher on Monday.

Local clergy have pointed out that residents who took issue with voting at the Islamic center could vote by absentee ballot, and that dozens of other religious institutions serve as polling sites in Palm Beach County, which is Florida’s largest. Five synagogues and around 90 churches will receive voters during this year’s elections.

TPM reached out to 16 churches in the area for comment, but pastors either did not respond, were out of town or declined to address the situation.

Rabbi Greg Weisman of Temple Beth El told TPM that many of his congregants expressed to him their support for the Islamic center and their desire for the elections supervisor to reverse her decision, although he said others disagreed.

A refugee crisis and recent terrorist attacks in Orlando and abroad have fostered an anti-Muslim backlash among some south Floridians. Rabbi Arele Gopin of the Chabad Lubavitch of Boca Raton said he has personally heard members of his congregation say they were “uncomfortable” voting at the Islamic center.

Keeping it as a polling site would “not be accommodating people’s right to vote,” he said.

At least some of that discomfort appears to stem from allegations directed at the center’s president, Alhalabi, who is an engineering professor at Florida Atlantic University. Video of Alhalabi explaining at an FAU panel on Islamophobia how Sharia law calls for thieves’ hands to be cut off has been widely circulated on conservative websites. He’s also been criticized for his association with a Palestinian-American civil rights activist who was deported following years of legal battles stemming from an indictment for allegedly providing “material support” to the Palestinian Islamic Jihad group. All charges against Sami Al-Arian, with whom Alhalabi co-authored technical papers as an engineering student at the University of South Florida, were ultimately dropped.

Alhalabi vigorously denied any links to Islamic extremism in an interview with TPM. He said he was explaining how Sharia law operated at the FAU panel and would never advocate for its application, and the university has stood behind him for years.

But some claim Alhalabi’s background renders his mosque unfit to receive voters. Upon learning that the Islamic Center would no longer serve as a polling site, the Florida Family Association pushed out an email template for followers to write to Bucher to express their approval for her decision. The Tampa-based, Christian conservative group splintered off from the American Family Association, which was labeled a hate group by the Southern Poverty Law Center.

One rabbi TPM spoke with expanded on why he thought some people bought into the accusations about Alhalabi and the center.

“The fact of the matter is that there are a lot of stupid people here,” he said, speaking on condition of anonymity. “There are lot of people who have allowed themselves to buy into the notion that all Muslims are bad and Sharia law is coming to take away our rights.”

David Steinhardt, a rabbi at Boca Raton’s B’nai Torah and a friend of Alhalabi’s, said that it was the responsibility of elected officials to counter such misconceptions.

“We have the capacity here to move things in a different direction,” Steinhardt told TPM. “Sometimes that takes a little bit of courage. Sometimes it’s a matter of simply doing what its right. If we believe in the values of American democracy and religious pluralism, then this decision is wrong.”