Freedom of movement in Europe has been all but abandoned as a cause in British politics. Brexit was far more about freedom of movement than our exact trading relationship with the EU, and the electorate rejected it. Even among remain voters, polling has suggested that more than half want increased border controls and European migration capped. Every demographic thinks there has been too much immigration over the last few years, even young voters, albeit far less so than their grandparents.

Faced with the reality of the referendum, Labour has a simple mantra. “Freedom of movement will end when we leave the European Union,” reads the manifesto, as though describing the weather. But it is a political choice, even if the alternative might seem to spell electoral doom. This isn’t to question the anti-racist politics of the Labour leadership: for the London Labour left, issues of race and defending migrants and refugees were always an article of faith, not least for the shadow home secretary, Diane Abbott. Jeremy Corbyn’s first act as Labour leader was to speak at a rally in defence of refugees, and speeches by leading figures emphasise challenging the scapegoating of the foreign-born.

Warning of 'utter chaos' if May ends EU free movement next March Read more

But here’s how I see it. The voters of the Stockport ward in which I grew up voted to leave the EU, some of my childhood friends and their parents among them. Immigration there is far below the national average. Yes, there are many social problems: a lack of affordable housing (because of the failure to build it), a lack of secure jobs (because they’ve been stripped from the economy), stagnating living standards (because of government policies), and struggling public services (because of austerity). Yet many of these problems were seen through the prism of migration, even in communities (such as my own) with very little actual migration.

Here is one consequence of the disappearance for a generation of a mass socialist movement – until its recent belated re-emergence – which would have offered a different narrative: that these are injustices caused by a broken system that has to be replaced. It is also the case that other working-class communities in big cities with high levels of migration as well as multiple social crises – not least high levels of poverty and a lack of affordable housing – voted to remain. In some, though not all, communities with higher levels of migration, anti-immigration sentiment is lowest.

For years, I’ve used my platform – including at the Guardian – to make the following arguments. Migrants are not responsible for the social ills and crimes of the powerful. Nurses, cleaners, builders, baristas and care workers from Poland, Lithuania or Spain did not crash the banks, did not avoid taxes on an industrial scale, did not systematically employ workers (wherever they are from) on poverty wages. Public services such as the NHS would collapse overnight without the foreign-born. It is not exactly an argument that passes the “explain that down the pub” test, but it is also true that numerous studies have shown that EU migrants pay more to the state than they get back. Yes, there is some evidence that migration can slightly depress wages at the bottom end of the labour market, but that’s an argument for a genuine living wage, for ensuring all workers are employed on the same terms and conditions, and for extending unionisation.

I believed that before the Brexit vote, and in those beliefs – to quote our prime minister – nothing has changed. It would be conceding all of the arguments I’ve made over migration; that those I so passionately opposed were actually right all along. And here’s my concern. The strategic decision to abandon freedom of movement inevitably means that the arguments against the scapegoating of migrants no longer have the public prominence they deserve. Bigoted and xenophobic lies long predated the EU referendum, of course, but the leave campaigns gave them renewed legitimacy, leaving many migrants feeling insecure, anxious – and abandoned, even.

It’s Labour who must have the difficult conversations on freedom of movement | Stella Creasy and Catherine West Read more

Yes, freedom of movement is only for Europeans, and the EU aggressively – violently, even – polices its own borders against non-EU migrants and refugees. But a rejection of freedom of movement within Europe’s own boundaries does not strengthen the case for accepting more migrants and refugees from outside: the reverse, in fact.

The case for freedom of movement may seem irrevocably lost. Many will argue that retaining it would make a mockery of the referendum result, and that public opinion is steadfastly against it, particularly among working-class voters. I understand these arguments. Labour may currently have no choice for political reasons. But the case surely has to be made, because the argument against is based on myths, however pervasive; and because the migrants who are our neighbours and friends, who prop up our public services and make so many contributions to our society, need and deserve our solidarity. Public opinion is not static: it can shift, but only if the arguments are made. And that’s why the case for freedom of movement must not be abandoned.

• Owen Jones is a Guardian columnist