The UK should stay inside the EU’s single market and customs union even if this means there can only be limited cuts to immigration after Brexit, the former Labour leader, Ed Miliband, said.

Spelling out his own Brexit strategy, and insisting that tough choices must now be made, Miliband argued that safeguarding the strength of the economy should be the number one priority in negotiations over leaving the EU, rather than a focus on the “undeliverable promise” of cutting immigration to tens of thousands a year.

While he accepted that there need to be changes to EU freedom of movement rules, as they apply to the UK, following the vote to leave, he argued that excessive concentration on cutting numbers will make a positive economic outcome for Britain more difficult to achieve.

Brexit must be fair to working people – or there will be a backlash |Ed Miliband Read more

This is because the EU will insist that any country that remains inside – or retains substantial access to – the single market must also accept the overall principle of free movement and that any reforms of its operation in practice should be limited.

After last week’s Commons vote, in which most Labour MPs voted in favour of Theresa May beginning exit talks by the end of March, Miliband said the “phoney war” is over and that tough choices and trade-offs will have to be agreed, which ministers have not prepared the public for.

His essential message is that the government will damage the country’s economy – and face a resulting backlash of huge proportions from the British people – unless it shows leadership and approaches negotiations in a realistic manner. “Government will rightly be subject to an almighty backlash from Leave voters, if it makes decisions that make them far poorer and leaves less money for public services,” he said. “Having voted for a better future, this would be the ultimate betrayal.”

“The reality is that there are very difficult trade-offs and risks that lie ahead, which the government has completely failed to acknowledge. It must now make choices to minimise those risks.”

Miliband’s warning comes amid increasing signs of division in both Tory and Labour ranks over what approach to take towards Brexit negotiations. Ministers have yet to make clear how they intend to achieve their stated aim of gaining maximum access to the single market, while at the same time wanting to restore full control of borders and break free of the European Court of Justice, which makes rulings on disputes and upholds single-market law.

Labour’s nuanced approach to Brexit – and divisions within the party over how tough it should be on immigration – was blamed by many for its disastrous result in last week’s Sleaford and North Hykeham byelection, when it dropped from second place to fourth behind both Ukip and the Lib Dems. The previous week, Labour lost its deposit in the Richmond byelection, as the Lib Dems stormed to victory with the backing of disillusioned Remain voters.

Labour frontbenchers, such as Clive Lewis, are calling for stricter controls on immigration, while others, including party leader Jeremy Corbyn and shadow home secretary Diane Abbott, insist that Labour should not be pushing for cuts in the number of EU migrants.

Amid evidence that a clear message is not getting through to voters, the party continues to plummet in the polls: on Friday, a YouGov survey put Labour down two points on just 25% – its lowest level of support in opposition since 1983. The Tories were up three points on 42%, a commanding 17% lead.

Facebook Twitter Pinterest Clive Lewis is calling for stricter controls on immigration. Photograph: David Gadd/Sportsphoto Ltd./Allstar

On Saturday, the Labour leader endured an uncomfortable afternoon when his speech was interrupted by Syria protesters, led by veteran campaigner Peter Tatchell, calling for Labour to take a stronger stand against Russian intervention in the Middle East.

Senior Labour figures said the party was trying to manage a growing divide between MPs and supporters in London and other big cities, who are broadly in favour of immigration, and those in more deprived regions, particularly in Labour heartlands in the north, who believe strict controls have to be part of Labour’s programme.

In a sign that the divide is causing growing problems for party management, three Labour frontbenchers – shadow foreign minister Catherine West (Hornsey and Wood Green), shadow education minister Tulip Siddiq (Hampstead and Kilburn) and shadow transport minister Daniel Zeichner (Cambridge) last week defied a three-line whip on a key Brexit vote in the Commons without apparently receiving any reprimand. All three represent cosmopolitan constituencies that voted overwhelmingly for Remain: West and Siddiq’s two London constituencies both voted 75% in favour of staying in the EU, while the figure in Cambridge was 74%.

Miliband cited an opinion poll by YouGov for Open Britain, released on Sunday, as evidence that the British people will not accept a Brexit deal that damages the economy and their own finances.

Poll suggests public will not accept a Brexit that leaves them worse off Read more

Half of those who voted to leave the EU in June, including 62% of Labour Leave voters and 59% of those in the north, would not be willing to lose any money at all as a consequence of Britain’s withdrawal. Just one in 10 would be willing to lose any more than £100 a month.

The poll also shows more than one in five (22%) of voters do not expect Brexit to have any impact on their finances at all – despite Vote Leave’s now infamous promise that quitting Brussels would add £350m a week to the public purse. Another poll, by consumer magazine Which?, indicates the economic effects of Brexit are a major worry for much of the public.

It shows almost half the population (47%) are worried about Brexit, up eight points since September, with nearly two-thirds concerned about its potential impact on food prices.