Before the last election I shared a train journey with Ed Miliband and some of his advisers on their way to a live television debate. The conversation turned to mugs. Labour had produced a range emblazoned with campaign pledges, and one offering “controls on immigration” had angered the left of the party.

Miliband was puzzled and frustrated. The promise to manage the nation’s borders followed logically from the impossibility of fighting an election on a manifesto implying the opposite. Who would advocate uncontrolled immigration? Besides, it was hardly the centrepiece of the campaign. Why get so hung up on one pledge when there were four others that the left should be cheering – protecting the NHS, for example, and raising living standards? There were mugs for those too.

But the problem wasn’t the mug so much as the ease with which an empty political vessel is filled with scepticism. No one who resented Labour for lax border policy in the past was impressed by a slogan of control; and anyone who found that kind of rhetoric distasteful was appalled that Labour should be borrowing it. One side doubted that Miliband meant what he said; the other side hoped he didn’t. He paid the maximum price in perceived inauthenticity, for no return in votes.

With immigration, perhaps more than any other policy area, the credentials of the messenger matter as much as the message. There is more space for consensus than is apparent in a polarised debate. Public attitude surveys find little support for the open-door approach, but also limited interest in raising drawbridges. Most people recognise that migrants make a valuable contribution to British society and always have done, but they also fret about the pressure put on public services – and, in some sectors, wages – by new arrivals. Many also find rapid change in the cultural contours of their neighbourhoods disorienting. Those are not unreasonable concerns, and the surest way to provoke mistrust is to conflate them with racist attitudes.

The point is made well and with reference to detailed polling in a report – published in 2014 but still fresh with insight – from British Future, a thinktank specialising in matters of migration and identity. The authors identify an “anxious middle”, comprising around half the population, who simultaneously value ethnic diversity and mourn a loss of cultural continuity. They are flanked by roughly equal 25% tranches of “migration liberals” on one side and “rejectionists” on the other.

Ed Miliband’s immigration mug. Photograph: Labour

A routine failure among politicians and commentators on the liberal wing is to treat the anxiety of the middle as a shade of rejectionism – either closet prejudice or a naive susceptibility to that tendency. Having diagnosed the problem as misinformation spread by the enemies of tolerance, they treat it with “myth-busting” cures, hoping that a blast of cold logic might dispel the miasma of false consciousness.

But to someone made anxious by personal experience, myth-busting comes across either as an accusation of stupidity (“you don’t understand the issues”) or a symptom of elite arrogance (“your unhappiness is a price worth paying for my continued economic success”).

With immigration, perhaps more than any policy area, the credentials of the messenger matter as much as the message

There is a blind spot on the other side too, exemplified by those Conservative Brexiteers who fought a campaign promising simple immigration solutions, and now plead for time, citing the complexity of their task. Boris Johnson, Liam Fox, David Davis – the cabinet ministers responsible for engineering a post-EU settlement of Britain’s relations – might be appalled at the suggestion they pander to racism, and yet this triumvirate owes its position to an electoral project that mobilised all the racists.

Of course not everyone who voted for Brexit is xenophobic, but xenophobes voted leave; and to pretend that the campaign did not give licence to casual bigotry, with an undercurrent of violent menace, defies decency and evidence.

Leave campaigners packaged up their answers to the border question as “an Australian-style points system”. That solution has now been rejected by Theresa May, not because it is too stringent but because the model is too lax, with too many avenues for automatic entry entitlement. So not really taking control at all. It was designed not to reduce numbers but to increase the population of Australia by importing labour and it succeeded in that aim, as the Brexit bunch knew all along.

The “points-based system” was always a phoney offer. It was clever shorthand for “non-racist yet rigorous-sounding alternative to the status quo”, chosen because it seemed meritocratic (points are earned) and culturally digestible (Australia is a friendly, Anglophone cousin country). It was no more viable as a device for containing public anxiety about immigration than Miliband’s mug, albeit with more retail appeal. Nor did it much improve on David Cameron’s commitment to cap net migrant numbers – a pledge with the dubious honour of being even more politically cynical and less practically attainable than George Osborne’s fiscal targets.

Thus are we confronted with the Conservative Eurosceptics, tongue-tied but blinking smugly in the glare of referendum victory, holding a blank piece of paper where they should have answers on immigration.

For years they posed as valiant crusaders against an oppressive liberal taboo. Countless times was it said that we couldn’t talk about immigration; that just to raise the concern invited nasty and unjustified insinuations of racism. Then it was said that there could be no solution to the problem without leaving the EU, that liberal elitists had propped open the doors and were using economic scare tactics against anyone who might close them.

Well, the field belongs to the sceptics now. The regime of the Bruxellois metropolitans is vanquished. They might once have had something sensible to say about the need to balance border control with economic openness, diversity and tolerance, but their message-carrying credentials are shot. The taboos they policed are all comprehensively broken.

So come on then, Johnson, Fox, Davis and friends: you can talk about immigration without being racist. It isn’t that hard. Start talking. Take control. Or does it turn out that this was just a meaningless slogan – in which case I’ve got a mug to sell you.