Immigration is not to blame for this country’s problems. This should be the agenda Labour is setting on immigration. One that speaks out in favour of freedom of movement and against anti-migrant sentiment, as shadow home secretary Diane Abbott has done in recent weeks. But at the moment, not everyone is on board – some risk conceding ground to the right on immigration. This is short-sighted and strategically a mistake: it further feeds the narrative on which the populist right thrive.

Since the referendum result, there has been a chorus of voices calling on Labour to give up on freedom of movement or abandon “diversity“. Last Sunday, on the Andrew Marr Show, shadow foreign secretary Emily Thornberry showed that there was a very real possibility that Labour would, yet again, capitulate to anti-migrant sentiment. She implied that migration drives down wages, which isn’t supported by the evidence, and said she thought there were too many people coming into this country. All of these ideas buy into the “us” and “them” message that contributes to a toxic immigration debate at the moment.

A significant number of Labour politicians and columnists are fond of resorting to a well-known piece of advice: the party must listen to peoples’ “legitimate concerns” about immigration. As Stephen Bush wrote recently, we should think about what those concerns actually are, while realising they can’t necessarily be boiled down to one decisive explanation. If the concerns are that migration drives down wages and has caused a crisis in public services, then none of this is borne out by the evidence. If peoples’ concerns are rooted in these fallacies, the best strategy is to explain that migration isn’t to blame. This doesn’t mean ignoring very real economic woes but it should translate into the party offering up plausible solutions. As journalist Rachel Shabi has argued, clamping down on freedom of movement is not the answer to low wages, the severe lack of housing or the crisis in public services.

If peoples’ concerns are that they don’t like multiculturalism, that they feel their national identity is being threatened or that there are too many people of colour living down the road, then Labour should speak to collapsed communities across the country armed with an anti-racist strategy that counters prejudice. This message will be likely take time to cut through but any other option is a dead end. We are witnessing the rise of ethno-nationalist sentiment in the UK; certain forms of “genuine” white identity are being closely tied to ideas of Britishness and Englishness. That’s why in the wake of Brexit, British-born people of colour – as well as migrants – were the subject of attack. Labour has to resist, argue against and counter this racist and xenophobic trend. And they have to do so with a genuinely honest message on migration.

But the idea that Labour should shape their response to the migration debate with evidence is, at times, met with ridicule. “You can’t just rely on facts,” is the regular response. But facts haven’t been present in the national conversation about migration for over 50 years: serving up the realities on immigration is not a tried, tested and failed strategy on the left. When the Tories have pushed an inflammatory anti-migrant message, Labour have submitted. At the last general election, Ed Miliband promised to clamp down on migrants claiming benefits despite the fact that most people weren’t claiming at all. One of his hallmark policies – which was stamped on the side of a mug for sale on the party’s website – promised “controls on immigration”.

What’s more, few pro-migration proponents are suggesting Labour only use one string in their bow. Infusing the immigration debate with facts doesn’t equate to relying on them and nothing else. And it needn’t result in a dry, unintelligible narrative rooted only in number. Putting facts front and centre of Labour’s message won’t win the argument but neither will throwing them out the window. A narrative that’s based on anti-migrant myths will only strengthen the right.

Labour should have a national pro-migrant message that’s paired with local campaigning. They should be in food banks and libraries; supplying services for people abandoned by the government. In the long term, the party needs to push for better education about race, racism and empire – without challenging widely-accepted stereotypes that foreigners are the problem, or contextualising those ideas, then it will be impossible to erode prejudice. There is no point in claiming that racism and xenophobia played no part in public attitudes towards immigration or the Brexit vote when the opposite is patently true.

Labour need to show a united front on immigration. MPs should be sticking to a narrative that explains living standards aren’t declining because of migration, that people from other countries aren’t a threat to the nation, while realising there is a strong race element to this debate. If Britain leaves the EU under an agreement that puts a halt to freedom of movement, we won’t see the end of anti-migrant sentiment. As the likely economic blow of reduced immigration takes full effect, the people who’ll be blamed will be migrants. Labour should realise this and unite around a credible narrative, which includes a pro-migrant message, now.