The clutch of England fans in Marseille were unequivocal. “Fuck off Europe, we’re all voting out,” they chanted. I’ve spent the week listening to much the same, politer, but just as fingers-in-the-ears adamant. No fact, no persuader penetrates their certainty – and these were Labour voters. Will Labour’s campaign week, kicked off by Gordon Brown in the face of a dire new Guardian poll, shift many outers?

Inside Labour’s London HQ, I joined young volunteers manning the “Labour In” phones with every fact at the ready. We had sheets of Labour-supporting names to call in Nottinghamshire – and the results were grim. “Out”, “Out” and “Out” in call after call, only a couple for remain. “I’ve been Labour all my life, but I’m for leave,” they said. Why? Always the same – immigrants first; that mythical £350m saving on money sent to Brussels second; “I want my country back” third. And then there is, “I don’t know ANYONE voting in.”

EU referendum: leave takes six-point lead in Guardian/ICM polls Read more

Try arguing with facts and you get nowhere. Warn these Labour people what a Johnson/Gove government would do and they don’t care. Warn about the loss of workers’ rights and they don’t listen – maybe that’s already irrelevant to millions in crap jobs such as at Uber or Sports Direct. “We’re full up. Sorry, there’s no room for more. Can’t get GP appointments, can’t get into our schools, no housing.” If you tell these Labour voters that’s because of Tory austerity cuts, still they blame “immigrants getting everything first”. Warn about a Brexit recession leading to far worse cuts and they just say, “Stop them coming, make room for our own first.”

Here were the two irreconcilable faces of Labour, eager young London graduates on the phone making scant headway with older traditional voters of Nottingham, impervious to love letters from Der Spiegel or heartfelt warnings that Labour people, Labour areas, would pay the price for Brexit self-destruction.

Every week in Barking the MP Margaret Hodge invites a whole ward for coffee and biscuits to air whatever’s on their minds. When the BNP shockingly won 12 council seats, those open-door meetings dealing with everyday grievances saw her make the case and beat them off, so the BNP lost every seat. On Friday about 50 voters came, wanting to talk about ordinary things – parking, fly-tipping and houses in multiple occupation crammed with migrants by rogue landlords. Hodge and her volunteers went from table to table recording everyone’s issues, writing to them later with resolutions.

But at the end when she asked the hall about the referendum, the mood changed. “We didn’t come to talk about that!” one angry woman said, others agreeing. “We came about parking!” But Hodge insisted, making an eloquent remain case: shrinking services are caused by Tory austerity that halved their council’s budget, more than migrants. The room bristled with antagonism. “Do you want to be governed by Brussels?” one shouted out. “You’re being sold a false prospectus, a bunch of lies,” she said, to no avail. One said: “When I get out at the station, I think I’m in another country. Labour opened the floodgates.”

They like her, a well-respected, diligent MP, but they weren’t listening. She demolished the £350m myth, but they clung to it. She told them housing shortages were due to Tory sell-offs and failure to build but a young man protested that he was falling further down the waiting list, with immigrants put first. Barking’s long-time residents come first, she said, but she was not believed. I found just two remainers.

This is Labour London, supposedly remain’s stronghold, though the Barking and Dagenham Post finds 67% for Brexit. Jeremy Corbyn is under fire for half-heartedness, but I doubt he, Brown or any Labour figure could budge them. Roused by anti-migrant leavers, will they ever revert to Labour? Their neighbourhoods have changed beyond recognition, without them being asked.

Children emerging from the primary school next door, almost all from ethnic minorities, are just a visible reminder for anyone seeking easy answers to genuine grievance. As high-status Ford jobs are swapped for low-paid warehouse work, indignation is diverted daily against migrants by the Mail, Sun, Sunday Times and the rest. “Fury over plot to let 1.5m Turks into Britain” was Monday’s latest from the Daily Mail.

This is the sound of Britain breaking.Here ends our “moderate, tolerant” self-image. Imagine Brexit wins and two years later prime minister Boris Johnson is still embroiled in quarrelsome EU exit talks. These Barking and Nottingham people will see no change, same migrants, same sense of powerlessness. Recession-hit, facing worse cuts, voters won’t blame themselves for their own folly. Old problems are unresolved – an economy reliant on City and property bubbles, low skills, low productivity, atrophied public services, hopes raised and dashed. Gove and Johnson risk losing control of the furies they have unleashed.

As high-status Ford jobs are swapped for low-paid warehouse work, indignation is diverted daily against migrants

That moment is fertile for some yet-worse demagogue who calls for throwing out migrants already here. Expect the volume to be raised against “elites” – anti-parliament, anti-politics, bored of democracy itself. Ignite hatred against Europe, blame Brussels for deliberately impoverishing us in revenge, stirring centuries-old enmities.

Blend all that with a little nationalistic leftish populism, not all of it bad: nationalise our utilities and rail, eject foreign owners from key industries and property, pump up armed forces and national pride. These are potent ingredients for militant majoritarianism, blaming minorities and minority opinions. The Human Rights Act is abolished and the BBC absorbed into government. National socialism will no doubt carry a new name – but it’s there in the making.

If remain scrapes in, David Cameron may urge the other 27 EU members towards some brakes on migration. After our near-death experience, with France’s Front National leader Marine Le Pen advancing, Poles and Hungarians screeching right and even worse threatened, some change looks necessary. Social democratic values, sharing within a community, both are threatened by an entirely open door.

Surely this can’t happen here? We’ll wake up from the nightmare on 24 June remembering we’re a moderate nation, unseduced by Pied Pipers of far right or left. Don’t count on it. We know how civilised democracies can be inflamed by racism and xenophobia. And whatever the result, where does all this anger go next?