The Israeli prime minister has stoked a volatile debate about refugees and migrant workers from Africa, warning that "illegal infiltrators flooding the country" were threatening the security and identity of the Jewish state.

"If we don't stop their entry, the problem that currently stands at 60,000 could grow to 600,000, and that threatens our existence as a Jewish and democratic state," Binyamin Netanyahu said at Sunday's cabinet meeting. "This phenomenon is very grave and threatens the social fabric of society, our national security and our national identity." Israel's population is 7.8 million.

His comments follow media reports of rising crime, including two gang rapes, in southern Tel Aviv, where many African migrants are concentrated. However, Micky Rosenfeld, spokesman for the Israeli police, said the overall crime rate in Israel had fallen. There had been one alleged rape of a teenage girl connected to the migrant community, for which three suspects were in custody, he added.

Yohanan Danino, the Israeli police chief, said migrants should be permitted to work to discourage petty crime. Nearly all are unable to work legally, and live in overcrowded and impoverished conditions. "The community needs to be supported in order to prevent economic and social problems," said Rosenfeld.

But the interior minister, Eli Yishai, rejected such a move, saying: "Why should we provide them with jobs? I'm sick of the bleeding hearts, including politicians. Jobs would settle them here, they'll make babies, and that offer will only result in hundreds of thousands more coming over here."

Yishai repeated an earlier call for all migrants to be jailed pending deportation. "I want everyone to be able to walk the streets without fear or trepidation ... The migrants are giving birth to hundreds of thousands, and the Zionist dream is dying," he told Army Radio. Last week he said most migrants were involved in criminal activity.

According to police data quoted by the Hotline for Migrant Workers, the crime rate among foreigners in Israel was 2.04% in 2010, compared with 4.99% among Israelis.

More than 13,500 people entered Israel illegally in 2010, of whom almost two-thirds were Eritrean and one-third were Sudanese. Three were granted refugee status by Israel, rising to six last year. Human rights organisations say more than 50,000 asylum seekers and migrants have entered Israel illegally since 2005.

Most are smuggled across the Israel-Egypt border by Bedouin tribesmen. Israel is constructing a vast steel fence through 150 miles of the Sinai desert as a deterrent to people-trafficking and the smuggling of drugs and weapons. The barrier would be completed, bar one small section, by October, Netanyahu said.

Israel is also constructing the world's largest detention centre for asylum seekers and illegal migrants, capable of holding 11,000 people. The £58m building, close to the border, will receive its first detainees by the end of the year.

Netanyahu said the state would embark on "the physical withdrawal" of migrants, despite fears among human rights organisations about the dangers they could face in their home countries. Yishai said: "I'm not responsible for what happens in Eritrea and Sudan, the UN is."

As tensions rise in cities with relatively high African populations, the past month has seen a spate of attacks on buildings in south Tel Aviv that house asylum seekers and migrant workers. In one incident, a Molotov cocktail was thrown into the courtyard of a kindergarten. NGOs working with migrants have also received abusive and threatening calls.

Amid the anti-immigration clamour, some Israelis have argued that, in the light of Jewish history, their state should be sympathetic and welcoming to those fleeing persecution.