Politicians should not put an arbitrary cap on migration because numbers fluctuate according to the health of the economy, Andy Burnham has said. In a Guardian debate on the EU, the shadow home secretary gave a firm defence of the role immigrants play in the NHS and dismissed the idea of a fixed limit.

“The Conservative party are realising it’s a mistake to put an arbitrary number on the whole issue. It is much more complicated than that,” he said. “The numbers will go up and down according the the current health of the economy. You could have an overall drive to bring down unskilled migration. You could do that by better protecting skilled workers.”

His comments came after the Labour leader, Jeremy Corbyn, said it was not possible to put an upper limit on migration within the EU.

Burnham has previously sounded a much tougher tone on immigration, arguing last year: “For too long, we have argued that free movement across Europe benefits everyone and affects all areas equally. That’s just not true … In places, a free market in labour benefits private companies more than people and communities. Labour hasn’t faced up to that, and that’s why we look out of touch.”

EU referendum: our panel on Jeremy Corbyn's Sky News appearance Read more

Burnham said he had been booed in some leadership meetings for saying there was an issue with immigration and that the whole political class was now “reaping the whirlwind” of the failure to talk about it. He said, however, that it was essential not to “ape the other side” and that Labour must discuss practical measures to relieve pressure on communities without resorting to what he called the xenophobia and discrimination of the leave campaign.

Labour is worried that many of its followers will vote to leave the EU and afterwards desert the party for failing to listen to their concerns about immigration.

Labour’s John Mann, who recently came out for the leave side, said he believed Labour’s core traditional working class vote would solidly opt to leave the EU and that the middle class would vote solidly to remain. The MP said the country was polarised and that it was an opportunity for the Labour party to “get on the front foot” and start representing working people again.

During the debate, Burnham and Mann also clashed over whether the EU protects the NHS from privatisation in one of the first “red on red” confrontations of the referendum debate. At one point, Mann accused Burnham, a former health secretary, of overseeing the privatisation of an ambulance service in his area under EU procurement rules.

The debate saw Burnham joined by the Lib Dem leader, Tim Farron, and the Green MP Caroline Lucas for remain, and Chris Grayling, the Tory leader of the Commons, and Daniel Hannan, a Eurosceptic Tory MEP, arguing with Mann to leave.

Grayling asked why the Green party was willing to see green space built over to accommodate immigration. She replied that migration goes both ways and argued that it was beneficial to society. Lucas said the “fury of the campaign had taken everyone by surprise” and that there needed to be a re-engagement with angry voters afterwards.

