Amid the winding, cobbled streets and art galleries of the holy city of Safed, an ancient stone house has become the centre of a seething tension that some say threatens to open up a new front in the conflict between Jews and Arabs.

In this house lives 89-year-old Eli Tzvieli – Holocaust survivor, former social worker and now scourge of the religious Jewish community around him. Next door lives Shmuel Eliyahu, the town's chief rabbi and leader of a campaign to drive out Tzvieli's tenants.

The old man rents rooms to three Israeli-Arab students. He has been threatened with having his home burned down, and notices have been pinned to his front door denouncing him as a traitor to Judaism.

Eliyahu, son of a former chief rabbi of Israel, advocates the expulsion of all Arabs from land he says God gave to the Jewish people. Now an Israeli cabinet minister is calling for Eliyahu to be removed from his post. Avishay Braverman, the minister for minority affairs, last week lodged a formal complaint with the justice ministry, saying Eliyahu's "continual incitement against the Arabs in the Galilee harms the fabric of relations between Jews and Arabs and does not serve the interests of the state".

The rabbi, along with 17 others, signed a letter this year ordering Jewish landlords not to rent to Arabs, saying: "Their way of life is different to that of Jews … [they] are bitter and hateful towards us." Neighbours should ostracise such a landlord, "refrain from doing business with him [and] deny him the right to read from the Torah".

A conference held last month under the banner of "Quiet War: Combating Assimilation in the Holy City of Safed" attracted 400 participants, including extreme rightwing activists.

Safed, which has a Jewish population of about 40,000, many of whom are conservative and ultra-orthodox, has attracted a growing number of students from Israeli-Arab villages to its college.

Their presence has led to accusations from Jewish residents – religious and secular – who speak of Arabs playing loud music on the Jewish sabbath, making approaches to Jewish women and causing fights with locals.

But Tzvieli, who spent time in a Hungarian labour camp during the second world war before moving to Israel in 1950, defends his tenants. "They are nice boys," he said. "People tend to generalise and it's wrong. Because we suffered from antisemitism, we should not act like antisemites." He used to get on well with his neighbour, he said, "but in the past few years [Eliyahu] started expressing extremist views".

One of Tzvieli's tenants, Nimran Grefat, 19, who is studying to be a nurse, said: "We feel threatened, we feel helpless. They don't want us here. We came to learn in peace, not to make war. Rabbi Eliyahu is a racist who hates Arabs." Eliyahu declined to be interviewed.

The town, one of four holy cities in Israel, had a mixed population until the 1948 war, when the Arabs were forced to flee. Now the residents –, many of whom emigrated from the US, are determined to preserve its Jewish character.

"Having Arabs on the streets changes the flavour of the city," said a woman who declined to give her name. "It's not comfortable to walk on the street – we have to dodge them all the time. When Jews move into Hebron or East Jerusalem, there's an international outcry. This is the same, but where is the outcry?"

Reuven Trabelsi, 31, a secular Jew who rents his cafe premises from Tzvieli, said his landlord was an honest man, "but I think he's wrong. Sometimes you need to look around, listen to what people say. This is a religious community."

The situation has echoes in other cities. The deputy mayor of nearby Karmiel was sacked this month for allegedly organising a "night patrol" to man roadblocks to keep Arabs out of the city.

Israel's extreme rightwing foreign minister, Avigdor Lieberman, has advocated the compulsory "transfer" from the country of Israeli-Arabs, who make up about 20% of the population. The government wants non-Jewish citizens to pledge loyalty to the Jewish state.

Tzvieli insists he will not bow to pressure over his tenants. "Most residents of Safed are a mixture of religious and secular and are tolerant and open," he said. "It's just here in the old city, where the extremists live, that I am in a minority."