David Cameron has suffered a blow to his authority after 97 MPs, many of them Conservative backbenchers, voted to curb the ability of foreign criminals to resist deportation on the grounds of their right to a family life.

The amendment, tabled by the rebel Tory backbencher Dominic Raab, was defeated by 241 to 97 votes after Labour and the Liberal Democrats voted against the measure.

Downing Street, which believed the amendment would put Britain in breach of its commitments under the European Convention on Human Rights, instructed its MPs to abstain to avoid a potentially crippling defeat.

The collapse in relations between No 10 and its backbenchers, which meant the prime minister was forced to rely on the Labour party to ensure Britain did not fall foul of Europe's human rights watchdog, prompted Chris Bryant to joke that the government "has become stateless". This was a pun on a separate government plan to potentially make foreign-born British citizens stateless if they present a threat to national security.

Yvette Cooper, the shadow home secretary, said: "The government's immigration bill has become a complete car crash. Tory ministers ran away from their own MPs today even though they told the Commons themselves this could have made it harder to deport foreign criminals.

"I can think of no precedent for government ministers abstaining on an amendment that they oppose because they are scared of their backbenchers. The home secretary told the house the measure was illegal and would make it harder to deport foreign criminals instead of easier. Yet she then refused to vote against it."

Downing Street played down the significance of the vote. Sources said Cameron had instructed his MPs to abstain because he sympathised with the principle behind the amendment but could not support it because it was not technically compliant with the law.

A source said: "We could not have voted in favour of the amendment because it was not workable. But if you vote against something you agree with in principle you send out a very strange signal."

The Tory whips told their MPs to abstain after the Commons speaker, John Bercow, called the amendment in the first batch of amendments to the immigration bill to be debated.

The amendment, which had won the signatures of 106 MPs, would have curbed the ability of judges to block the deportation of foreign criminals, and prevent them from resisting deportation on the grounds of their right to a family life except when it would cause "manifest and overwhelming harm" to children.

Theresa May had hoped to reduce support for the Raab amendment by proposing to strip foreign-born terror suspects of British citizenship if they were judged to present a threat to national security. It would even apply to those who had no other citizenship, rendering them stateless.

But rebels dismissed this as a displacement exercise.

May told MPs: "Depriving people of their citizenship is a serious matter. It is one of the most serious sanctions a state can take against a person and therefore not an issue that I take lightly."

The move, which was branded by the human rights organisation Liberty as "irresponsible and unjust", is supported by the Liberal Democrat leader, Nick Clegg.

He said the current laws had become a "passport for endless games in the courts to prevent people being deported that should be. We are tightening up the way the courts can interpret article 8, the right to a family life, so it cannot become an excuse for unjustified legal procrastination."

Speaking on LBC radio's Call Clegg show, he said he knew the plan was controversial, but justifiable in a very small number of cases. The revocation of British citizenship "would apply in cases where individuals pose a real threat to the security of this country".

Shami Chakrabarti, the director of Liberty, said: "Liberty always said that terror suspects should be charged and tried. First, politicians avoided trials for foreign nationals; now they seek the same for their own citizens.

"This move is as irresponsible as it is unjust. It would allow British governments to dump dangerous people on the international community, but equally to punish potential innocent political dissenters without charge or trial. There is the edge of populist madness and then the abyss."

The government decided to act after the supreme court ruled last year that a terror suspect accused of targeting British troops had the right to return to Britain from Turkey after being stripped of his British passport, which he gained in 2000. The supreme court restored UK citizenship to Hilal al-Jedda, who was born in Iraq, on the grounds that he would be stateless without it.