We have learned that the remain campaign’s attempt to prevent a flood of Labour voters backing Brexit will have new faces. The relaunch began on Monday on Radio 4’s Today programme, when Gordon Brown was interviewed by John Humphrys. It did not go well – and not just because Brown became irritated with the way most of the questions were about immigration. It didn’t go well because immigration is the issue that explains why a third or more of Labour voters might back Brexit.

The Labour party finds itself in the very dangerous position of having only 4% of its MPs supporting leave but, in one opinion poll, 44% of its members doing so. My pro-EU colleagues in the parliamentary party should not expect to have any material effect on this huge gap if the messenger changes but the message stays the same.

People who already have wealth, own a home of their own and have a secure job may not worry about immigration, but Labour’s heartland supporters do worry about it. If present trends continue, two-thirds of the projected increase in population over the next 25 years is expected to come from immigration. Ten to 16 million extra people will move to Britain, suggests Migration Watch, and place huge demands on our schools, hospitals, roads and housing stock. That increase might be manageable in good economic times but it could become very socially divisive when, inevitably, the economy hits bumpier times.

It’s not racist to worry about this, as some of my colleagues seem to think, and there’s broad support for more control of immigration among all Britain’s ethnic minority communities.

Gordon Brown at a pro-EU rally on 9 June in Glasgow. ‘My pro-EU colleagues in the parliamentary party should not expect to have any effect on this gap if the messenger changes but the message stays the same.’ Photograph: Jane Barlow/PA

Leaving the EU doesn’t mean an end to immigration but it does mean that we will be able to decide who comes here and how they come. We must still welcome the dedicated medical professionals who help keep our NHS on track. We can still admit the entrepreneurial and highly qualified individuals who will help build prosperity. If immigration is controlled and people begin to have faith in the system again, I also hope we might be open to taking more refugees from the world’s trouble spots. In other words a post-EU immigration regime can support our public services, expand our economy and also deliver humanitarian objectives; but because it will be under our control there won’t be unexpected and excessive pressures on our schools, hospitals and public infrastructure.

And as Barbara Castle, that great Labour campaigner of yesteryear once argued, outside the EU we will no longer be discriminating in favour of Germans, Spaniards or Belgians and against Indians, Australians and Canadians as is required by our membership of the EU. Britain can become a proper citizen of the whole world again through a colour-blind immigration policy, and not one that serves a “little Europe” and only our closest geographical neighbours.

I also wish my Labour colleagues would have more pride in our party’s reforming history and the good sense of the British people. It was Labour governments that introduced the vast majority of the rights that workers enjoy.

From holiday pay to the minimum wage, Clement Attlee, Harold Wilson and Tony Blair didn’t need European bureaucrats to impose them on us. Again and again, Britain’s social rights have been world-beating. I’m absolutely confident that if any Tory government was foolish to attack our fundamental rights they would fail in their ambitions to gain support in the north of England and in the cities.

I’m struck by the ways in which this government is actually trying to copy Labour and steal our clothes on workers’ rights. The national living wage and the apprenticeship levy are half-baked attempts by David Cameron and George Osborne to enact the kind of changes that I hope the next Labour government will implement properly.

With youth unemployment still close to 50% in many of our continent’s southern neighbours – notably in Greece, Spain, Italy and Portugal – it is obvious that Europe isn’t working for workers any more. Millions of Labour’s voters have noticed and are voting to leave the EU as a result. They’ve heard politicians say that Brussels is about to change for the better if only we are all patient once more, and they know it never does change.

Our voters are wiser on this big question than Labour’s leaders, and I hope if Brexit does occur on 23 June a “freedom flame” will start to burn across the continent and social democratic parties, trade unionists and social justice campaigners will rise up against the existing EU model. It’s a model that suits Goldman Sachs and big businesses who want cheap labour. It suits agricultural industries rather than the families shopping for groceries. It’s not a social Europe. It’s not a workers’ Europe. It’s not a modern Europe. It’s time to leave.