Some years ago I joined a delegation of Labour MPs to the European parliament. A member of the EU commission gave us some of his valuable time. “The British people are just going to have to learn to put up with freedom of movement,” he said as he sat down.

Everyone around me agreed. I remember thinking that the British people didn’t have to put up with anything, not their politicians and certainly not their bureaucrats. But I said nothing. As a recently elected MP I did not have the confidence to interrupt the consensus. And besides, like all my colleagues, I supported freedom of movement.

That consensus is now breaking down. Earlier in the summer I wrote about the fears of many Jeremy Corbyn supporters that without him Labour would race not simply to the right but to a fusion of Blairite managerialism and Farage populism.

I thought those fears unjustified, a Corbynista version of Project Fear. But some are now arguing that, like Brexit, Project Fear is becoming Project Fact as Labour MPs argue for the party to walk away from freedom of movement and the home secretary proposes lists of “foreign” employees.

It’s not only in the UK. Across Europe the rise of the right has made comrades question the appetite for freedom of movement. While the tone and actual proposals differ, the message is often similar. The white working class has been ignored for too long. We need to check our privilege and check out of Europe.

I don’t know that much about privilege. I grew up like the white working class, with one obvious difference: I wasn’t white. I’m certainly in a position of immense privilege now, and that gives me a voice when it comes to my values, but it did not help form them. The white working class helped form my values.

On North Kenton council estate, where I grew up, racism was a daily occurrence that marked me for life. But I also benefited from a huge amount of support from working-class friends, families and teachers. I learned abiding and entrenched views on fairness, the value of labour and the importance of solidarity. The values that caused Abraham Lincoln to thank “the working men of Manchester” for their solidarity against slavery. The values that formed both the co-operative movement and the fair trade movement.

When I was selected as candidate for Newcastle Central I said there was no doorstep in the city where I would not go. If someone was voting BNP or Ukip, I would listen and find out why. I learned that we have a huge job to do to regain the trust of many who have been betrayed by rising inequality, reduced demand for traditional skills and increased competition for certain types of work. There is real and justified anger at that betrayal.

I can see that such anger might incline some comrades towards if not a flirtation with xenophobia then at least a wary two-step (“Look, we’re not touching, just moving to the mood music”). They hope in so doing to shore up Labour’s position as the party that speaks for the working class, the same reasoning that led the party to abstain on the second reading of the welfare bill last year. This latest dance is likely to be just as successful, but the impact on our reputation would be even worse.

There are certainly arguments to be had about freedom of movement of people versus freedom of movement of exploitation and welfare. And there is much to be said for knowing what happens at our borders, and who comes in to the country and who is leaving it.

But these arguments must be made from a position of strong support for free movement of people, the jobs it brings through access to the European market and the value those people add to our economy and our society. Without the free movement of people, the free movement of goods and capital isolates and dehumanises us, free trade without free people both commoditises our experience and makes us less resilient to global forces of change.

This is toxic territory. It does not lend itself to “triangulation” – particularly by those who have little familiarity with the working class they claim to represent. For too many, the working-class experience is an essay in anthropology, not a reality to be reflected – far less challenged or questioned.

And even if we could triangulate it to try to win more votes, we should not. Labour party members have for the second time overwhelmingly demonstrated they want authenticity. On this subject the Tories can authentically outplay us every time, as this week has shown.

Ultimately no matter how many soundbites we drop, Labour is not going to be believed as the party that is tough on immigration. We can be believed as the party that is tough on the negative consequences of migration. But if we let our values dance to the tune of xenophobia, we will not be believed on anything.