(JTA) — On a recent Sunday, eight far-right activists filmed themselves on what they called a “raid” on the Aurora Jewish community center in Budapest.

Sporting crewcuts and black clothes, the men affixed posters with a crossed-out picture of the Hungary-born Jewish American billionaire George Soros to the entrance of the building, which along with having a Masorti, or Conservative, synagogue, also serves as the headquarters for a gay rights group, a Roma advocacy lobby, a hotline for immigrants and several other nongovernmental organizations with liberal agendas.

The activists then spray-painted the words “Stop Operation Soros” on the sidewalk opposite Aurora’s front door.

“Time permitting, we will say hello again,” read an article published about the action on May 2 on the website of the ultranationalist Sixty-Four Counties Youth Movement. The menacing article did not mention Jews, but did note the Roma and gay rights activity at Aurora, which it falsely claimed is financed by Soros.

To Adam Schonberger, a Hungarian Jew who runs Aurora, the raid was an anti-Semitic attack arising from a campaign that is being led in billboards, television ads and speeches by Hungary’s right-wing government against Soros. In recent months Prime Minister Viktor Orban, who is critical of the European Union’s federalist and progressive agendas, has flagged as “dangerous” Soros’ promotion of those plans by funding grassroots activism and some higher education frameworks in Hungary.

Occurring in a conservative society that is still struggling with the complicity of its wartime governments in the murder of nearly half a million Jews during the Holocaust, the campaign against a Jewish billionaire has prompted warnings that Orban’s crusade against Soros is anti-Semitic. Earlier this month Frans Timmermans, a senior EU official, suggested that the Hungarian government is channeling anti-Semitic sentiment to delegitimize a powerful critic of its nationalist policies.

That view, however, is not shared by the main leaders of Hungary’s 100,000-strong Jewish community. In interviews with JTA, its leaders rejected allegations that the government is using anti-Semitic dog whistles consciously. At the same time, they warned that the campaign against Soros may embolden anti-Semites regardless of the government’s intentions.

“Orban is not anti-Semitic. His government is not anti-Semitic,” said Rabbi Zoltan Radnoti, the chairman of the rabbinical council of the Mazsihisz Jewish umbrella group in Hungary. “I believe that Soros was selected as a target because he is a progressive billionaire regardless of the fact that he’s Jewish.”

Yet Orban failed to stop the anti-Soros campaign even after it appeared that the rhetoric “may have a possible anti-Semitic interpretation,” Radnoti added, saying the prime minister “should have known that this campaign of hatred and scapegoating would increase anti-Semitic feelings.”

Soros, an 86-year-old banking and investment magnate who survived the Holocaust in hiding in Budapest, is not particularly known for funding Jewish causes in Hungary — or anywhere else.

In Hungary, Soros has given away approximately $400 million since the early 1990s — part of a $12 billion expenditure, according to the Open Society Foundations, which is Soros’ network of organizations throughout Eastern and Central Europe. Much of the money has gone to supporting progressive causes, including the promotion of minorities and multiculturalism. But other programs offered tuition to needy students – including Orban, who studied at Oxford in 1989 on a scholarship provided by a Soros-funded organization.

Soros is best known in Hungary for opening the Central European University, which he founded in 1991. Orban’s government is promoting legislation that could lead to its closure.

The conflict between Soros and Orban, who have long tried to avoid a collision, escalated in 2015 when the prime minister clashed publicly with leaders of the European Union over Hungary’s refusal to take or let in hundreds of thousands of immigrants, including refugees, from the war-torn Middle East.

Under attack from Brussels over the migrant crisis, as well as his crackdown on foreign funding from Norway and beyond for Hungarian nongovernmental organizations, Orban lashed out at Soros, accusing him of trying to flood Europe with foreigners. Soros hit back with a defiant statement that won him some acclaim, but also deepened resentment toward him on a continent where Muslim immigration and extremism is leading far-right parties to unprecedented gains.

“Our plan treats the protection of refugees as the objective and national borders as the obstacle,” Soros said.

Upping the ante, Orban gave a speech last month at the European Parliament calling Soros a “financial speculator” who is now “attacking Hungary and who — despite ruining the lives of millions of European people with his financial speculations” is nonetheless “received by the EU’s top leaders.” The scathing rhetoric was followed by the appearance in Hungary of posters demonizing Soros, which are widely believed to be printed and distributed by nationalists with the government’s blessing.

And that’s a problem, according to Radnoti, because it risks awakening anti-Semitic sentiments that Radnoti believes Orban neither shares nor seeks to embolden.

“The problem is not that Soros was selected as a public enemy because he is Jewish,” Radnoti said. “The problem is that in a country like Hungary, which has a xenophobia and anti-Semitism problem, the government should have known better than to take someone who happens to be Jewish and make him a public enemy over his globalist politics. It’s not anti-Semitic, it’s just irresponsible.”

Compared to the Hungarian Jewish leaders, Timmermans went too far, said Janos Gado, an editor of the Hungarian Jewish monthly Szombat.

“The leftist, Social Democrat EU politicians are failing to understand the nature of the new right wing to which Orban belongs,” he said. “It’s not dictatorial, it’s authoritarian. It’s not racist, but it is illiberal. So they refer to it in anti-fascist terminology that frankly does not apply to the situation.”

Notwithstanding, Soros has inspired much anti-Semitic rhetoric in Hungary and beyond. In Poland, a far-right nationalist at an anti-immigration rally in Poland set fire to an effigy of an Orthodox Jew that he later said represented Soros. And in March, the American radio host Alex Jones, a Donald Trump supporter, ranted about “the Jewish mafia” that he said was run by Soros.

But Hungary has not seen any significant increase in anti-Semitic rhetoric or incidents since the escalation of the fight between Orban and Soros, according to Daniel Bodnar, chairman of the board of the TEV watchdog on anti-Semitism in the country. The group’s annual report, which was published this month, noted surveys suggesting hostility toward Jews has not increased this year.

“The government’s motivations are definitely not anti-Semitc,” Slomo Koves, the leader of the Chabad-affiliated EMIH Jewish group, told JTA. “So when Jewish community leaders inject themselves into the debate about Soros, they are in a sense appropriating his politics and associating them with the Jewish community. Not only is this a misrepresentation of the spectrum of opinions held by Hungarian Jews, it is also a dangerous game.”

While many Hungarian Jews who support liberal causes view Soros favorably, other Hungarian Jews take issue with some of his positions and actions – including his funding for groups seen as anti-Israel.

In addition to funding Israel-based organizations critical of their country’s policies — notably Breaking the Silence and B’Tselem — Soros’ Open Society Foundations have donated millions of dollars to groups that NGO Monitor, a right-leaning group in Israel, calls anti-Israel. They include Al-Haq and Al Mezan, Palestinian groups that promote boycotts against Israel.

Hungarian Jews were widely critical when Soros’ Central European University extended an invitation in 2015 to Joseph Massad, a Palestinian academic from Columbia University who has said Israel does not have a right to exist as a Jewish state.

“It was absolutely scandalous,” said Laszlo Seres, a Jewish journalist for the Heti Világgazdaság who has written critically about Soros, calling him a “self-hating Jew.”

(In 1995, Soros wrote that “I am proud of being a Jew,” but added that in the past he had “suffered from the low self-esteem that is the bane of the assimilationist Jew.”)

Yet even Soros’ Jewish critics feel uncomfortable in the face of the government-led campaign against him, Seres said.

“First of all, Soros despite all the problems connected to the man has donated to a lot of worthy causes in Hungary,” Seres told JTA. ”And the government’s hysteria around him, well, it’s ugly and counterproductive. It makes it very difficult to criticize him.”