There never used to be a problem with the free movement of people until it became the mass free movement of people.

It’s not the principle that people should be able to travel and work in other EU member states that’s wrong – it’s that the scale at which they are doing so.

Voters’ concerns about immigration are often just worries about the effects on wages, housing and public services. They have tried to raise them at various elections in the past few years, but the political class hasn’t been listening.

They hear politicians talk about the “perception” that mass migration is a problem, but they never admit that it actually is. This is why they telling us they will vote to leave the EU next week. We have to listen. But we have to act too.

In recent days, various Labour figures – Ed Balls, Tom Watson and Yvette Cooper – have been trying to do that, raising the prospect that we will look again at how free movement works in practice in order to protect our own workers.

We should start by saying that there is now too big a gap between what was intended when free movement rules were drawn up in the 1950s and what happens now in practice.

It’s a bit like the row about US gun laws. The freedom to “bear arms” shouldn’t mean people have the right to own an AK-47. The original intention and how it manifests itself today have become completely detached.

It’s like that with free movement. The EU’s founding fathers created a system in the 1950s where small numbers of professionals could work in different countries. It wasn’t something that millions of people did as a routine, as indeed they do today.

If you work in a non-unionised, low-paid, insecure, private sector job, probably on a zero-hours contract, you know what it’s like to face low-cost competition.

We know there are unscrupulous employers who only recruit overseas, working with gang masters to undermine the minimum wage and workplace protections that we, in the Labour party, have fought so hard to put in place.

Despite his promises, David Cameron’s EU renegotiation didn’t deliver the goods in terms of reforming the free movement rules. It’s meant this referendum campaign has become completely dominated by immigration. But rather than trying to ignore the issue, I believe we should face it head-on.

So how can we go about addressing voters’ concerns about free movement and the impacts of mass migration at this late stage?

First, the European Commission needs to send a signal that it recognises this is an issue of concern, not just for Britain, but for people in many other member states too. This issue doesn’t begin and end with this country. The Commission needs to go back to the drawing board and come up with a system that doesn’t result in the poor and non-unionised looking over their shoulders, fearful that they will face competition for what little they have.

That’s not the social Europe we should be building.

Second, we need to see far more attention paid to so-called gang masters and those exploiting migrant workers. This is a matter for national governments. There should be much tighter regulation of our labour market, especially in those industries that we know use cheap migrant labour, where people are often treated as little more than slaves.

We can start by making it easier to establish a trade union presence in sectors like food production where workers’ rights are virtually unheard of.

Third, we need a proper Living Wage that puts a meaningful floor under workers’ wages and stops the recruitment of cheaper foreign workers becoming such an attractive option. And we need tough enforcement so that any employer not paying it faces much heavier penalties. These should include being struck off as a company director and in the worst cases, custodial sentences.

Likewise, we need to see the new apprenticeship levy, which come in next May, cover more employers. Currently, it will not affect any business with a wage bill of less than £3m. This should be lowered, ensuring there is no place to hide when it comes to businesses paying their fair share for training our own workers.

As a lifelong trade unionist, I know you need to be in the boardroom to negotiate the deal you want. There is no point shouting from behind a closed door. This is what Brexit would mean. The EU is not the perfect solution, but neither is leaving. If we stay in, we can fix the problems around free movement and work together with other member states who are becoming just as concerned about the failings of free movement as we are.

And it’s safe to say that politicians from the Prime Minister down cannot fail to have heard the message these past few weeks. At last, they are starting to get it.

Let’s not cut our nose off to spite our face.

Joe Anderson is Labour Mayor of Liverpool