(CNN) Hull lies on the muddy banks of the River Humber in northern England. It is often called the 'Gateway to Europe.'

Every morning, ferries unload their cargo from the Netherlands and Belgium. And every evening, trucks loaded with goods are shipped across the North Sea to mainland Europe.

But the 'Gateway to Europe' is a glamorous title for a long-neglected city.

For many, life is hard. Parts of this community are among the poorest and most deprived in the whole of the United Kingdom.

And the town showed last week that they're among the most fed up with the European Union.

Some 68% of Kingston upon Hull residents voted for a Brexit -- that's a ratio of more than two to one.

'Nothing to lose'

Most people here are happy to tell you why they voted out.

We met Christine and Gary Bristow outside the shopping center on the Orchard Estate, one of the town's poorest areas.

A photo posted by Antonia Mortensen (@antoniamortensencnn) on Jun 27, 2016 at 9:13am PDT

The Bristows are in their sixties and have lived in Hull their whole lives. They told us they were "fed up, there is nothing round here," and said they voted to leave because they feel left behind by Westminster.

They don't think that the European Union has ever done much for their area and blame the EU for the decline of the fishing industry in Hull. It was a thriving business when the UK joined the EU in the 1970s, but now is pretty much non-existent.

Many residents we spoke to in Hull shared the same view; they backed a Brexit because they felt things were so bad they had nothing to lose.

Pensioner Cecil Fordham voted out because he believes social services are over burdened by the town's newcomers.

"There's your doctors isn't there. You can never get there. You're two weeks waiting to get in to your doctors," he said.

"Nowt [nothing] to lose have we. I don't think so... we ain't got nowt [nothing] anyway."

The blame game

Gary Cole, 44, was born on the Orchard Estate and voted to remain.

A photo posted by Antonia Mortensen (@antoniamortensencnn) on Jun 27, 2016 at 9:19am PDT

"I'm furious to know that people don't know all the facts and are just finding them out now, it gets me very angry. I've got a daughter at university who will come out with thousands of pounds of debt. She's 18, she could have gone abroad."

When asked why he thought so many people voted for a Brexit in Hull, he said, "people like us, no one is listening to us, that's why they voted out."

This is a common view we heard again and again -- that the local result was punishment for the policies of British politicians, especially the ruling Conservative party.

For many, there is also a sense things have changed too quickly for the worse, and they blame immigration.

Jacko Paul, who voted out, also lives on the estate: "It doesn't seem to be safe anymore. It was always a really close-knit community, but now [there are] too many gangs of Eastern Europeans going around."

Susan Lightfoot was born in Hull, and she voted in.

A photo posted by Antonia Mortensen (@antoniamortensencnn) on Jun 27, 2016 at 8:51am PDT

"Too much is happening in the world for us to come out. Nobody knows what is happening now, whether it's good or bad, and we don't know where we are going."

Susan also thinks a lot of people in Hull voted to leave because they are worried about immigration.

Made to feel unwelcome

Driving down the city's Beverley Road, we pass a string of Polish "Polski Sklep" and Eastern European shops.

JUST WATCHED What are the issues that led UK voters to leave the EU? Replay More Videos ... MUST WATCH What are the issues that led UK voters to leave the EU? 03:14

A Polish beauty salon owner, who wanted to remain anonymous, told us that she has been in Hull for 10 years and was shocked by the result.

"I am very worried for my business, most of my clients are Eastern European -- the British don't seem to trust our techniques," she said.

She said she would not leave, as her two children were born in the UK, and it would be too difficult for them to reintegrate back into Polish society.

Another Polish shopper told us that he works mainly with English people, and that some of them had stopped talking to her after the vote.

If it continued he was going to pack up and go home, as he did not want to stay in a country where he was made to feel unwelcome.

No belief, no benefits

Angus Young covers politics for Hull's local newspaper and said he was surprised at the vote.

He said recent European immigration has energized parts of the economy, with shops opening on formerly abandoned streets. But there have been social strains too.

"There has been a big influx. And it's caused a lot of tension and it's displaced a lot of people. People are feeling uneasy. [It's] put a lot of pressure on services."

There is new money coming into Hull -- private, public and from the EU.

German company Siemens and Associated British Ports are investing more than three hundred million pounds in a plant for making offshore wind turbine blades.

Some here are predicting jobs and growth.

But too many believe they will never see the benefits.