Europe needs to restore borders and impose controls on economic migration if it is to stop extremist political forces exploiting anti-refugee prejudice, the former shadow chancellor Ed Balls has said.

Writing in the Wall Street Journal, Balls said David Cameron had achieved “the helpful goal” of limiting benefits paid to migrant workers, but that it was only a matter of time before European leaders revisited the rules on free movement of labour across the EU.

“If Europe does not eventually agree to restore borders and impose controls on economic migration, the initiative will pass to populist forces on the far left and right whose aims are not to manage globalisation fairly, but to exploit prejudice and rig markets in favour of homegrown producers,” he said.

Balls’s comments come on the day the prime minister is to make a speech at the annual St Matthew’s Day banquet in Hamburg setting out the wider agenda behind his renegotiation of Britain’s membership of the EU before a national referendum. Cameron will attend a crunch summit in Brussels next Thursday and Friday to further those negotiations.

Balls, who lost his seat in May’s general election, said: “Those necessary EU changes will not come quickly enough for Britain’s referendum, but that is all the more reason why Britain must retain its influence in Europe to fight and win these arguments in the years to come.”

The scale of migration by east European workers had been the biggest driver of anti-EU sentiment in the UK and the refugee crisis was “pouring fuel on that fire”, said Balls, who was economic secretary to the Treasury in Tony Blair’s government. He is currently a senior fellow at Harvard University’s Kennedy school.

The “woeful failure of the European single currency” had made it understandable that some people wanted to leave the EU, he said. “As the eurozone lurches from crisis to crisis, many British voters understandably – but wrongly – think Britain would be better off leaving the EU entirely.

“Leaving the EU would both weaken Britain’s voice on the big global issues and damage the nation’s economy. With half of UK trade tied up in Europe, and many US and Asian multinationals currently basing their European headquarters in London, walking away now would cost us investment, jobs and income, as well as influence.”