Brussels has launched a ferocious counterattack against David Cameron over immigration, saying his talk of benefit tourism and a something-for-nothing culture among EU migrants is unintelligent and risks stoking "knee-jerk xenophobia".

Responding to Cameron's speech last week in which he pledged to restrict access to housing benefits and the NHS for those coming to the UK under EU free-movement rules, the European commissioner for employment, social affairs and inclusion, László Andor, told the Observer that his claims were misleading and very unfortunate.

"There is a serious risk of pandering to knee-jerk xenophobia," he said. "Blaming poor people or migrants for hardships at the time of economic crisis is not entirely unknown, but it is not intelligent politics in my view.

"I think it would be more responsible to confront mistaken perceptions about immigration from other EU countries and so-called 'benefit tourism', and instead to explain the facts.

"The reality is that migrants from other EU countries are very beneficial to the UK's economy, notably because they help to address skills shortages and pay more tax and social security contributions per head, and get fewer benefits, than UK workers; that free movement of workers is a key part of the EU's single market; that hundreds of thousands of UK nationals work in other EU countries."

The hard-hitting response suggests Cameron, who pledged to tighten residency tests for migrants from the EU and the wider European Economic Area, is heading for a bruising confrontation with Brussels if and when he tries to secure agreement for such an approach with other member states.

European law says that before EU citizens who are "not active in the labour market" become eligible for social security benefits, they have to pass a strict "habitual residence test" proving that they have a genuine link with the UK. Brussels says that the tests are stringent at all levels and that existing rules, agreed by the UK, prevent benefit tourism.

The current tests are already the subject of legal disagreements between the UK government and the commission, which believes there are cases in which the UK is already breaching EU law.

Former European commissioner and Labour cabinet minister Lord Mandelson – now co-president of a cross-party campaign to promote a positive role for the UK in Europe called British Influence – also rejected the idea that citizens from EU states come here to "sponge". "The truth is that Britain needed workers from other EU states when economic times were good, and we will probably need to attract them again," Mandelson said. "This isn't sponging, it's contributing. And of course it's two-way – roughly the same number of Brits went to work in other EU states."

In his speech, widely seen as a response to the rise of Ukip, Cameron said that "ending the something-for-nothing culture is something that needs to apply in the immigration system as well as in the welfare system". He pledged that "by the end of this year and before the controls on Bulgarians and Romanians are lifted, we are going to strengthen the test that determines which migrants can access benefits".

He also said: "We're going to give migrants from the EEA – from the European Economic Area – a very clear message. Just like British citizens, there is no absolute right to unemployment benefit. The clue is in the title: jobseeker's allowance is only available to those who are genuinely seeking a job."

Andor said the commission had asked the British government on many occasions for evidence to back its claims about benefit tourism by EU citizens but had received none. "The UK government has already been talking to us about so-called 'benefit tourism' for a couple of years, but whenever we have asked them for proof about the phenomenon they have been unable to provide it, despite repeated requests."

He said fears of a surge in benefit tourism after restrictions on the entry of Romanians and Bulgarians are lifted in January were unfounded. "People come to the UK from other EU countries to work, not to claim benefits. And we do not expect this pattern to change after 1 January, from when Romanian and Bulgarian nationals will also be free to work anywhere in the EU. Terms such as a 'something for nothing culture' are misleading and very unfortunate."

Studies have found that migrants from the 10 countries that joined the EU in 2004 - Cyprus, the Czech Republic, Estonia, Hungary, Latvia, Lithuania, Malta, Poland, Slovakia and Slovenia - have been 60% less likely to receive benefits in this country than UK-born residents and are 58% less likely to live in social housing.