Downing Street has rushed to defend a major speech by David Cameron on immigration as Labour claimed that it was "unravelling" after No 10 struggled to back up the prime minister's claims with hard statistics.

As the Joint Council for the Welfare of Immigrants (JCWI) accused Cameron of increasing intolerance, No 10 said the speech was designed to be a "forward-looking" attempt to "reduce pull factors" to Britain ahead of the granting of full EU cross-border working rights to Bulgarian and Romanian citizens next January.

Downing Street ran into trouble on a number of points the prime minister made in his speech.

Cameron said he was going to give migrants from the European Economic Area (EEA) a "very clear message" that there would be no absolute right to unemployment benefit. The EEA includes all 27 members of the EU plus Iceland, Liechtenstein and Norway. "The clue is in the title," Cameron said. "Jobseeker's allowance is only available for those who are genuinely seeking a job. You will be subject to full conditionality and work search requirements and you will have to show you are genuinely seeking employment. If you fail that test, you will lose your benefit."

But of the 2 million net migrants to the UK from the eight eastern European countries that joined the EU in 2004, just 13,000 people have claimed jobseeker's allowance (JSA). This figure was not disputed by No 10.

Downing Street also said there had been a 40% increase in the number of social lettings taken up by migrants between 2007-08 and 2011-12. This appeared to gloss over the fact that it was an increase from 6.5% to 9% in the proportion of such lettings.

The prime minister's spokesman cited the 40% increase to support Cameron's claim in his speech that he would end the "culture of something for nothing" on social housing by encouraging councils to give locals priority over housing. But this will also have to apply to UK citizens as well to comply with EU law.

"New migrants should not expect to be given a home on arrival," Cameron said. "And yet at present almost one in 10 new social lettings go to foreign nationals."

Labour pointed out that Mark Prisk, the housing minister, said in a parliamentary written answer in December: "Most foreign nationals who have recently come to England are not eligible for an allocation of social housing. Broadly speaking, European Economic Area nationals are eligible if they are working, self-sufficient, or have a permanent right of residence in the UK (after five years' lawful residence in the UK). Other foreign nationals are not eligible for social housing unless they have been granted leave to enter or remain in the UK with recourse to public funds (for example, people granted refugee status or humanitarian protection). Where foreign nationals are eligible, they will have their housing needs considered on the same basis as other applicants in accordance with the local authority's allocation scheme."

The prime minister also said that Britain has a "free National Health Service not a free International Health Service".

This claim was based on Department of Health figures that show that in 2011-12 the NHS was owed £33m by foreign nationals for hospital treatment. The NHS collected only £21m of this and the department estimates that it was owed at least double the £33m, giving a figure of £66m. This represents 0.06% of the £104bn NHS budget.

Jeremy Hunt, the health secretary, said he believed foreign nationals owed the NHS at least £200m a year, or 0.2% of the NHS budget. The Department of Health said the £200m figure was "speculation" based on a report by CCI insurance, published in 2003, which estimated the cost at between £50m and £200m.

In response to an urgent question in the Commons Hunt later unveiled plans to limit free NHS care to permanent, not temporary, overseas residents. He told MPs: "The initial phase of [a government] review has concluded and we will shortly start a consultation on a range of options, including plans to extend charging to some visitors and temporary residents who were previously exempt so that the default qualification for free NHS care would be permanent, not temporary, residence; ending free access to primary care for all visitors and tourists; introducing a prepayment or insurance requirement for temporary visitors to pay for NHS healthcare; and improving how the NHS can identify, charge and recover charges where they should apply. We will retain exemptions for emergency treatment and public health issues."

Yvette Cooper, the shadow home secretary, said: "The government's proposals announced today seem to be very confused and are unravelling. And at the same time there was no significant action to tackle illegal immigration or labour market exploitation which we know have been getting worse. We won't support an arms race on immigration rhetoric, we want practical and sensible measures that make the system work. We didn't get that from the prime minister today."

Downing Street said the speech was designed to set a template for the future. A No 10 source said: "Part of it [the speech] is on the importance of reducing pull factors from people who may be considering coming. So it is not just necessarily relating to people who are already here. There is a forward-looking angle to the speech."

In his speech the prime minister said it was important to act on immigration as he set out plans to restrict access to benefits for immigrants from the EEA and beyond. He announced, as expected, that JSA would only be available to those genuinely seeking a job for a maximum of six months.

He also announced that EEA immigrants would have to show a decent command of English. Cameron said: "We're going to make that assessment a real and robust one, and yes, it's going to include whether your ability to speak English is a barrier to work. And to migrants who are in work but then lose their jobs the same rules will apply. Six months, and then if you can't show you have a genuine chance of getting a job, benefits will be cut off.

"This means that EEA migrants who don't have a genuine chance of getting work after six months will lose their right to access certain benefits. So yes, they can still come and stay here if they want to, but the British taxpayer will not go on endlessly paying for them any more."

The prime minister was scathing about Labour's record. "Under the previous government immigration was far too high and badly out of control. Net migration needs to come down radically from hundreds of thousands a year to just tens of thousands.

"And as we bring net migration down so we must also make sure that Britain continues to benefit from it. That means ensuring that those who do come here are the brightest and the best, the people we really need with the skills and entrepreneurial talent to create the British jobs and growth that will help us to win in the global race."

Habib Rahman, chief executive of the JCWI, said: "This rhetoric may curtail rights to benefits on a minor scale, but relatively few migrants compared with 'indigenous' people actually claim benefit anyway. The real effect of this speech will be to further increase the intolerance and the hostile reception that immigrants are facing from some sections of society.

"There's nothing new about people from these countries coming to live and work in the UK. This media hysteria denies the fact that immigration helps our economy and is a great boon to tackling the coming demographic imbalance posed by our ageing population."