During the 1964 election Harold Wilson spent a day campaigning in London marginals, addressing crowds from the back of a lorry. Invariably he would be harangued by bigots demanding the repatriation of nonwhite people. Wilson faced the hecklers down. “Whom should we send home? The nurses in our hospitals? The people who drive our buses. Where would our health service be without the black workers who keep it going?” According to the late Paul Foot: “These questions were greeted with great roars of approval from the crowd, and the hecklers were silenced.”

Elsewhere that year a notorious election campaign in Smethwick, near Birmingham, saw the Tory candidate, Peter Griffiths, slug his way to victory on an anti-immigration ticket buoyed by the slogan: “If you want a nigger for a neighbour vote Labour.” When asked to disown that sentiment Griffiths replied: “I would not condemn anyone who said that. I regard it as a manifestation of popular feeling.”

Labour won the election nationally, with a 3.5% swing, but lost in Smethwick because of a 7.2% swing against them. Later, in his diaries, the Labour minister Richard Crossman concluded that since Smethwick: “It has been quite clear [for Labour] that immigration can be the greatest potential vote-loser for the Labour party”.

For the last 50 years the British political class has refused to engage intelligently with the issue of immigration. The Tories brazenly stoke popular prejudice (Margaret Thatcher: “swamped by people with a different culture”; Michael Howard: “Are you thinking what we’re thinking?”) while Labour cravenly submits to it (Tony Blair’s bulldog; Ed Miliband’s mug).

Wary of making arguments that are moral or fact-based, Labour sought not to counter inflammatory rhetoric but to indulge it. The Tories understand that fear of immigration is how they get votes; Labour understand that’s how they lose them. The upshot is that precious few in the country understand what immigration is for, what drives it, or who benefits from it and why.

Neither party wanted to pay the electoral price for engaging with the issue head on, or to offer a progressive counterpoint to prejudice. Now, with just a week to go before the Brexit referendum, those debts are coming due. Anti-immigration sentiment, across supporters of all parties, is proving to be the achilles heel of the remain campaign, which is backed by every major UK political party except Ukip yet is now losing in the polls.

For years, politicians on both sides kept saying it was time for an “open” and “honest” discussion about immigration. But there is no sensible debate on the subject you can have without talking about international trade and development, war, environmental catastrophe and global economic inequalities.

So long as most of the world lives on less than $10 a day then the poor will move in search of food and work. So long as we either displace or impoverish people through war, unfair trade and unethical arms sales, we will be in part responsible for people having to move. Moreover, unless we discuss the needs of an ageing population – the number of people over 85 is predicted to nearly treble in Britain over the next 30 years – and the net benefits that migrants make to our tax coffers, the conversations will never be complete.

Every discussion about foreign policy, international development, economic growth, labour regulation, the European social fund and climate change should also be an integral part of the argument about immigration. Being in or out of the European Union won’t shift those basic dynamics. It might make a difference to who comes, and how they get here, but it won’t make much difference to the general flow – which is dictated by demand for labour at home and, to some extent, desperation of labour abroad.

We’ve been talking about little else for quite some time. The trouble is, the conversation has been anything but open or honest. Nor has it been well informed. Three-quarters of Britons think immigration should be reduced. That’s hardly surprising. They think migrants comprise 31% of the UK’s population; the actual number is 13%. If you think something’s twice the size it really is, you’re bound to find it frightening.

But it’s not just about numbers; it’s also about resources. Britain opened its borders to migrants from eastern Europe in 2004. Four years later there was a massive financial crash followed by economic austerity. Nobody was held responsible for that calamity. The Tories made the poor pay for it instead – slashing services, closing libraries and cutting benefits.

In the absence of any major party campaigning against austerity or for reining in finance capital, some blamed the “strangers” in their midst – be they Roma, refugees or eastern Europeans – who stand as proxies for the anxieties prompted by neoliberal globalisation.

Herein lies the most obscene perversion of this turn in the Brexit campaign. The very people who are slashing resources – the Tory right – and diverting what’s left to the wealthy are the ones rallying the poor by blaming migrants for the lack of resources. Not content with urinating on our leg and telling us it’s raining, they have found someone to blame for the weather.