David Cameron has stepped back from a radical plan to cap directly the number of EU migrants entering Britain after an intervention from the German chancellor, Angela Merkel, who warned him she would not tolerate such an incursion into the principle of the free movement of workers.

The decision to row back from the harder rhetoric in a long-awaited speech on immigration on Friday disappointed the prime minister’s more Eurosceptic backbenchers, but delighted business leaders. However he still faces the task of persuading 27 other governments to change EU treaties to enshrine discrimination against European citizens working in Britain.

Downing Street officials said they were confident Cameron’s plan to deny EU migrants access to all in-work benefits for four years is negotiable and would deter tens of thousands from moving to Britain.

In the speech in Staffordshire on Friday, billed as one of the most important of his premiership, the prime minister also announced he would deprive EU migrants in work from accessing social housing for four years, and would not allow unemployed migrants to stay for more than six weeks.

Cameron stepped back from a bolder plan for an annual quota or an emergency brake apparently only in the last week following Merkel’s intervention. She said on Friday after Cameron’s speech: “The German government has in the past again and again underlined the significance of the principle of the free movement as it is anchored in the EU treaties. It is important that Cameron commits himself to this central pillar of the EU and the single market.”

Cameron was also being advised that he needed to shift the political agenda away from Europe to the economy, and setting up a massive confrontation with Merkel would only keep immigration at the top of the agenda, thus benefiting Ukip.

In a speech that repeatedly emphasised the benefits of UK membership of the EU, Cameron nevertheless tried also to emphasise his determination to secure his negotiating goal saying: “I don’t want to fail, I don’t believe I am going to fail. But to put it beyond doubt I am saying today that if I do fail, I rule absolutely nothing out, and I mean nothing”.

Asked if the plans would require treaty change, he replied: “The answer to the question is yes. These changes, taken together, they will require some treaty changes. There’s a debate in Europe about exactly which bits of legislation, which bits of the treaty you’ll need to change, but there’s no doubt this package as a whole will require some treaty change and I’m confident that we can negotiate that.”

But opening the treaties up to negotiation is a nightmare for most EU governments and there will be strong resentment if they are reopened just to suit Britain, or even to provide it with an opt-out.

EU common social security rules underpinning the freedom of labour regime make citizens equal wherever they reside and work within the EU. European commission vice-president Valdis Dombrovskis, a former long-serving Latvian PM, said: “A Latvian in the UK is entitled to social benefits if he is working”.

The Polish prime minister, Ewa Kopacz, one of the European leaders to whom Cameron spoke ahead of his speech, said: “The status quo should not be changed.”

The German news magazine Der Spiegel called Cameron’s speech blackmail.

Immigrants from the Baltic states such as Latvia and from Poland would be particularly affected by the policy shift outlined by Cameron. Dombrovskis said Cameron was on much safer territory with his drive to stamp out alleged abuses of the social security and welfare systems by immigrants, but the fundamentals of the EU could not be changed.

“It is the clear position of the European commission that the four freedoms [of goods, services, capital, and labour] are non-negotiable. They are part of the fundamentals of the EU ... If a person comes to a member state and is working or legally seeking work, he is entitled to benefits. That is the basis.”

Donald Tusk, the former Polish prime minister with whom Cameron has clashed over alleged “benefits tourism”, takes over on Monday as president of the European council chairing EU summits. He will have his hands full with the UK attempt to overhaul the terms of its EU membership.

“Tusk’s advice would be not to go necessarily in the direction of treaty changes as at end of the day it could be counterproductive,” said a senior EU official who will be involved in the negotiations.

Downing Street’s confidence about being able to force a re-opening of the Lisbon Treaty to facilitate the changes to free movement rules was not shared in Brussels.

“It’s difficult to imagine there would just be a treaty change about the UK,” said the EU official source. “Remember that decisions at the European council require consensus, something that has to be factored into proposals addressing UK concerns.”

This means that any of the 27 other governments could veto the British gambit.

But Cameron said: “What I am saying today is negotiable. It’s doable and what is more it will work. If Europe says no people will want a pretty good explanation why and frankly so will Our concerns are not outlandish or unreasonable. . We deserve to be heard, and we must be heard. Not only for Britain’s sake, but for the rest of Europe.

“Because what is happening in Britain is not unique to Britain. Across the European Union, issues of migration are causing real concern.”

He said EU migrants working in Britain with a couple of children at home are currently able to get effectively £8,000 a year of in-work welfare. “Removing that economic incentive is the most powerful thing we can do to reduce levels of immigration back to what British people want to see,” he said. The proposed reform would in essence remove the UK public subsidy of low wages that British businesses pay to EU workers.

Bill Cash, a veteran Eurosceptic Tory MP, said: “I do not think this goes far enough.”

Liam Fox, the former defence secretary said UK problems with the EU were not just focused on immigration. He said: “There’s the whole issue of the euro, the instability of the eurozone, and the economic threat it poses to the United Kingdom.”The deputy prime minister, Nick Clegg, had put forward a similar but less draconian package a few days earlier. He said : “I think some of the ideas from David Cameron are sensible and workable. There are some very serious question marks about whether others will ever really happen in practice and whether they are deliverable.”