David Cameron's plan to stop EU "benefit tourism" by restricting migrants' legal right to freedom of movement is running into serious difficulties after a parliamentary committee and the European Commission both accused ministers of failing to provide substantial evidence of widespread abuse.

At the same time, a joint report by the Centre for Economic and Business Research and global recruitment consultancy Harvey Nash has concluded that any measures to restrict freedom of movement to the UK from other parts of the EU would have a devastating effect on the British economy, because EU citizens are more likely to be in work and are more productive than British-born workers. The report says that curbing immigration from other parts of the EU could cost the UK £60bn in lost GDP (2% in real terms) by 2050.

News that ministers are struggling to make the case for tighter controls is a severe blow to Cameron, who wants the changes to be part of a new UK-EU relationship that he can put to the British people in an in/out referendum on EU membership by 2017.

Earlier this month Cameron suggested that there could be a way of restricting entry to "working immigrants" – even if this meant undermining the EU's key principle of "free movement".

He said: "Could the whole problem of immigration [and] welfare tourism be part of making sure we have a European relationship that works for Britain? Yes of course it can."

He added: "We should consider all of these things. I think we particularly need to look at the rules on benefits. One of the advantages of British membership of the EU is that British people go and live and work in other countries. But I think there is a problem with people living [here] and not working."

However, a House of Lords committee whose job is to scrutinise the government's EU policy, has written to the immigration minister, Mark Harper, saying that despite several requests the government has failed to offer anything more than anecdotal claims to back its assertion that "social benefit tourism" is a real problem.

In a statement to the Observer, the European commissioner for employment and social affairs, László Andor, also said that the UK was not proving its arguments with evidence. "It is important that policy on labour migration within the EU is based on facts rather than gut feelings or perceptions," he said. "If there are indeed substantial problems, the commission would of course be willing to consider how best to address them. We have not received any substantive data so far."

According to the CEBR/Harvey Nash report, immigrant workers are more likely to be in work (63%) than UK-born citizens (56.2%) and are more economically active. It says British businesses rely heavily on immigrant workers. Between 2003 and 2013, the number of non-British EU-born citizens employed in the UK went up from 762,000 to 1,647,000.

Albert Ellis, chief executive of the Harvey Nash Group, said: "Non-UK EU-born workers are bringing much-needed skills and value to the UK and there is little evidence that EU immigrants are having a negative impact on wages or unemployment. In fact, immigrants are helping to create jobs. A broad and diverse labour market fuels growth, as this report shows."

But a Home Office spokesperson said: "Free movement abuse is a genuine problem and a growing number of EU member states now share our concerns. We have joined forces with Germany, Austria and the Netherlands to demand the European Commission takes action."

The spokesman told of a recurring problem in London with groups of EU nationals who "arrive in the city, set up camps in public areas and beg and steal from tourists".