Outside Calais town hall, Pascale Malfoy and her daughter Coralie Condette were having a coffee near the flower-filled park. At the table next to them, three migrants were hunched over a sheet of paper, talking quietly. "There seems to be more of them than ever," said Condette. "Before they were kind of concentrated in one place, but now they are just everywhere."

Many in this port town of France feel embattled. And following incidents this week in which nearly 100 migrants attempted to storm a tourist ferry to Dover, they warn that if the situation does not improve, tensions could overflow and the far-right Front National (FN) could gain a stronger foothold in the town.

Ahead of a protest planned by far-right group Sauvons Calais (Save Calais) on Sunday, hundreds of migrants and leftwing activists on Friday marched through the town. In a calm, if rather desperate demonstration, they held banners calling for their human rights to be recognised, and chanted for "an end to police violence". By far the loudest shout came as a man with a microphone chanted: "We want to get to UK."

Holding a banner with the slogan Stop Beating Us, Nasser, 40, from Yemen said migrants were living in squalid conditions. "Why do people think I am here? I'm not here for a holiday," he said. "There are many problems in my country. Everyone wants a job, everyone wants money to support their family. This is why I have to get to the UK."

People in this town are used to a large migrant presence, but residents say the situation is more desperate than ever. Since the start of the year the numbers of migrants, which had dwindled, have dramatically increased to an estimated 1,000-1,400 living in the town and its environs. According to the local préfecture, 7,414 undocumented migrants were arrested in Calais in the first six months of this year, compared with the 3,129 detained in the same period of 2013.

In a cafe on Place d'Armes, Bernard Costenoble, a retired lawyer and judge, said that despite compassion for the migrants, frustration was mounting. "It's like a pressure cooker," he said. "If the steam can't escape … well, it explodes."

His friend Bernard Fleuet worries about growing support for the FN, fronted by Marine Le Pen. In European elections in May the party won 34% of the vote in Pas-de-Calais, one of its best tallies and a tripling of its score from the 2009 EU election.

"People here are not extremists, we are quite passive – but I am fearful that something is going to happen," he said. "The number of migrants is going up, the opportunities to get to the UK are fewer – there's a real risk of violence."

According to Sauvons Calais Sunday's protest will "say no to the invasion of Calais, of France and of Europe". It is a message that resonates with a growing number of young people, said Steve, 32, who did not want to give his surname. "Of course support is growing for the FN. Governments from the left or right have done nothing. At least with the FN you know there will be a dramatic change. People have had enough."

According to the UNHCR, worldwide refugee numbers are at their highest point since the second world war. While the vast majority of refugees stay close to home or in neighbouring countries, in Europe, there has been an estimated 70,000 migrants since the start of the year, said Franck Duvell, migration expert at Compas at Oxford University. While EU numbers have reached 150,000 in recent years, migrants are more concentrated because the number of routes has dropped which has brought the focus on to towns like Calais.

"There has been a lot of finger-pointing by different countries, but what is obvious is that this is a collective EU problem – and it is far from being addressed as one," he said.

The debate over who should take responsibility for the growing number of migrants is a secondary concern to those who have reached Calais, many from Sudan, Eritrea and Somalia. Alongside the roads to the port a relentless game of cat and mouse involves migrants trying to sneak in the back of any stationary lorry, only to be instantly shooed away.

On Friday in a patch of scrappy forest on the edge of town – the "jungle" – supposedly cleared in 2009, and again in May this year, a group of Afghans are making chapatis and tea around a campfire in their makeshift camp, and say they have no choice but to keep on trying.

Rahmat, 21, has lived in limbo in Europe since fleeing his country aged 15. He has been deported from Norway back to his entry point, Greece. But the next day, he resumed his journey.

"I have tried all European countries but no one has helped me," he said. "I have to make my future now in the UK. Maybe the UK people will help."

'It is hard to live here, but it is hard to live in Eritrea'

Sara sits on a wall along a disused railway on the outskirts of Calais, the hood of her black jacket pulled tightly around her face. The 23-year-old is wearing jeans, dirty socks and flip flops. She does not smile often, but when she does her teeth are startlingly white.

Sara explains that she made her way to Calais from Eritrea, and has been here about two weeks. Over a period of about three months, she journeyed overland into Libya where she waited to board a rickety open boat to Lampedusa.

Asked why she left, her smiles fades. "Everyone here has a different reason, for me I was a nurse but I wanted to study maybe to become a doctor - but there was no hope for me in Eritrea. Life is too hard."

Wanting to avoid being conscripted into the country's army, she decided to flee, saving money so she could pay her way at each stage of the journey. "I don't pay one person, I pay for the boat maybe 500 euros," she says.

Sara's family remain in Eritrea, she set off on this journey across continents completely alone and now sleeps in a makeshift tent in an area of woodland just outside Calais known as the jungle. She seems resigned to her situation, living without running water, food, showers or toilets.

"Eritreans help each other, we are like brothers and sisters," she says. "Yes, it is hard to live here, but it is hard to live in Eritrea."

Does she think it would be better in the UK. The smile returns, briefly. "Yes, I hope so," she says.