Under the flowering jacaranda trees in Tel Aviv's Levinsky park, clusters of men lounge on the grass smoking cigarettes, making desultory conversation or dozing. A few perch on children's swings, some push battered bicycles. A whiff of stale urine hangs in the air.

Beneath the restless boredom, there is a new mood: wariness. Since an anti-migrant demonstration last week turned ugly and violent, these men – many of whom fled in fear from their homes in sub-Saharan Africa – are fearful once again. The display of naked racism in what is routinely described as a diverse and liberal city also shocked many Israelis.

Around 1,000 people marched through an area of south Tel Aviv in which migrants are concentrated, chanting "blacks out" and holding signs calling for expulsion. Shops serving the migrant community were smashed up and looted; dark-skinned men were threatened. A member of the Israeli parliament told the crowd: "The Sudanese are a cancer in our body."

The protest followed a spate of reports in the Israeli media of a "crime wave" – including alleged rapes, assaults and robberies – blamed on migrants. Just a few days earlier the prime minister, Binyamin Netanyahu, had warned that a flood of "illegal infiltrators" was threatening Israel's identity.

In recent weeks, Molotov cocktails have been thrown at apartments housing African migrants. One firebomb was hurled into the yard of a kindergarten.

"I've never seen anything like it before," says Oscar Olivier, 45, who fled from Kinshasa 18 years ago. "It was a surprise, but in some ways it was also foreseeable. Some politicians have been orchestrating this, inciting violence and hatred, for months."

He was now very frightened, not just for himself but also his nine-year-old daughter, who was born in Israel but has no legal rights.

A straw poll in and around Levinsky park finds men from Sudan, Eritrea, Congo, Nigeria and Somalia, most of whom describe escaping "bad places" in search of a better, safer life. Generally they pay Bedouin tribesmen to smuggle them across the Israel-Egypt border. Some bring their families, but male migrants easily outnumber women.

According to Israel's population and immigration authority, 62,000 people have crossed the southern border since 2006. "If we do not stop the entry, the problem … could easily grow to 600,000 illegal infiltrators," Netanyahu said last week.

Israel is constructing a vast steel fence along the border to deter people-smuggling, due to be completed by the end of this year, but some analysts suggest the route will simply transfer to the border with Jordan, which is still open.

Once on Israeli territory, migrants are detained while their identity and country of origin is checked. Most are then given "conditional release" and put on buses to Tel Aviv, where they are dropped in Levinsky park and left to fend for themselves, or other Israeli towns and cities.

Their temporary permits to stay in Israel need to be renewed every three or four months, and specifically exclude permission to work. Many end up being employed on a casual basis for a pittance, living in overcrowded rundown apartments, confined to the fringes of society.

Some turn to crime out of desperation. "Yes, a few people have committed crimes, but not because they are foreigners but because they are hungry and not allowed to work," says Olivier. "You don't blame the whole community because of a few crimes."

But the arrest of three Eritreans in connection with the alleged rape of a Jewish teenager fuelled a backlash among Israelis living in or close to areas with a high migrant population.

The police chief Yohanan Danino called for migrants to be allowed to work to discourage petty crime caused by economic hardship. The interior minister, Eli Yishai, responded in uncompromising tone: "Why should we provide them with jobs? … Jobs would settle them here, they will make babies, and that offer will only result in hundreds of thousands more coming over here." Instead, he said, all migrants should be jailed pending deportation.

The Israeli state is anxious to avoid creating a sense of permanence among migrants. Last week, the US state department noted in a global report on human rights that in 2011, Israel had approved just one asylum application out of 4,603 made during the course of the year.

And in the coming week, the state is seeking court approval to deport at least 700 migrants to south Sudan, despite concern about their safety.

The question of how to respond to the influx of Africans has caused some anguish in a state founded by refugees and immigrants. But it has been swiftly drowned out by much louder voices denouncing bleeding heart liberalism among those who live in middle-class areas, far from migrant communities.

Many in Levinsky park simply feel they have swapped one kind of oppression for another. "They don't like black people in Israel," said Aldheer Ahmed, 45, from Darfur. Nasser Omeh, 24, also from Darfur, says reaching Israel was a positive moment in a bleak life. "But now they just want us to go away."

After leaving Congo, Olivier went first to Egypt, where he realised he had "left one dictatorship for another". He decided to head for Israel, "the only democracy in the region".

"I expected to be welcomed by people who understand what it is to be a fugitive. Now each day I am waiting for the violence and hatred to flare up again," he said.

Immigration to Israel

• 62,000 people have illegally crossed into Israel from Egypt since 2006, according to the population and immigration ministry.

• The vast majority originate from Eritrea (two-thirds) and Sudan (one-third).

• Most pay Bedouin smugglers $1,000-$2,500 to guide them across the border.

• Israel is constructing a 150-mile steel fence along the border to deter what it calls "infiltration".

• It is also building the world's largest detention centre, capable of holding 11,000 people.

• Israel granted asylum status to one out of 4,603 applicants last year, according to the US state department.

• Israel's population is 7.8 million.