David Cameron has been warned by German chancellor Angela Merkel that she would rather see the UK leave the European Union than change freedom of movement rules, according to reports.

Downing Street on Sunday did not deny that the conversation had taken place, after German newspaper Der Spiegel said Merkel had rejected Cameron’s demands for a cap on unskilled migrants. Sources told the newspaper that the chancellor said demands for any changes to freedom of movement rules represented a “point of no return” and that this would be it for the UK’s membership.

Over the weekend, the Sunday Times also reported that the prime minister has dropped plans for quotas in a bid to placate the Germans and that Cameron is now looking at whether the government can ask EU immigrants to leave the country unless they can support themselves within three months of arriving in the UK.

Asked whether the Der Spiegel report was an accurate account of the encounter between Cameron and Merkel, a Number 10 official would only say: “The prime minister will do what is right for Britain as he has repeatedly made clear.”

However, a spokesman for the German chancellor indirectly confirmed Der Spiegel’s report by reiterating a statement Merkel had made at a press conference after the recent EU summit: “Germany does not want to touch the basic principle of free movement of persons within the EU.”

Merkel’s reported arguments were backed up on Sunday by Ken Clarke, the senior Tory MP and former cabinet minister, who said free movement of labour was “absolutely essential” to the whole concept of a single market.

He told the BBC’s Sunday Politics: “Not only is that one of the principles of the EU, it’s the basis of any serious single market. The Norwegians, who eurosceptics admire, they accept the free movement of labour. They’ve got a bigger proportion of other EU nationals in their country compared with their own nationals than we have.

“If you want to be in the single market, if you want to be a modern economy then the free movement of labour is essential, but free movement of labour shouldn’t be abused.”

It comes amid speculation that Cameron is rowing back from his focus and tough language on immigration amid fears that the Conservatives will never be able to go as far as Ukip supporters want. Jim Messina, a US election strategist who worked on Barack Obama’s campaign, is said to have told a Tory awayday on Friday that every moment the party is not talking about the economy between now and the election is wasted.

However, Boris Johnson will keep to the subject of immigration on Monday, as he backs a report by the Commonwealth Exchange calling for more immigration from Commonwealth countries instead of so much from the EU.

In his foreword, he will say: “We should welcome the brightest and the best from a wider range of countries.

“As we re-examine our relationship with the EU, we have a vital opportunity to recast our immigration system in just this way. And the first place to start is with the Commonwealth.”

The issue of immigration is also tricky for Labour. Over the weekend, Ian Austin, a former aide to Gordon Brown and MP for Dudley North, called on his party to “be honest” and “say sorry” for opening the borders to eastern Europeans in 2004.

The row is the latest piece of Europe-related bad news for Cameron in recent days: last week he suffered a double blow when a parliamentary bill to establish an EU membership referendum by the end of 2017 collapsed, and a key UK ally told Britain to accept a demand from Brussels to pay an extra £1.7bn.