David Cameron's plan to fight the terrorist threat posed by Islamic State got off to a stuttering start when he was forced to shelve key proposals amid legal uncertainty, Liberal Democrat objections, and even doubts within the security services.

The prime minister unveiled a package of anti-terror measures in the Commons on Monday but was not able to include a widely trailed proposal to prevent British-born citizens returning to the country from Syria or Iraq if they were suspected of being involved in acts of terror.

Acknowledging the legal difficulties in preventing British citizens returning to the UK, admitting that it might render them stateless, the prime minister said new measures were still needed to prevent British jihadis returning.

He told MPs: "It is abhorrent that people who declare their allegiance elsewhere can return to the United Kingdom and pose a threat to our national security.

"We are clear in principle that what we need is a targeted, discretionary power to allow us to exclude British nationals from the UK."

But the prime minister's clear statement of intent was not backed by any proposals to match the rhetoric of Friday, when he used a Downing Street press conference to warn of the dangers of the "generational struggle" posed by the emergence of Islamic State (Isis).

The difficulties facing Cameron were underlined by the former Conservative attorney general Dominic Grieve, who warned that removing passports from UK-born citizens returning home would breach international law and UK common law.

Grieve said "even taking such powers on a temporary basis is likely to be a non starter".

It is thought Grieve gave this advice privately as attorney general last year when ministers agreed to take powers to remove passports from naturalised UK citizens that had the prospect of being a citizen of another state. He said the best course was to prosecute suspected terrorists in the UK courts.

Cameron said as many as 500 UK citizens were fighting in Syria or northern Iraq, representing the single greatest threat to the UK.

"It absolutely sticks in the craw that someone can go from this country to Syria, declare jihad, make all sorts of plans to start doing us damage and then contemplate returning to Britain having declared their allegiance to another state."

With the US launching air attacks against Islamic State recently, Cameron refused to give any firm commitment on UK involvement, saying no such requests had been received – a formula being used by ministers until a clearer strategy emerges from Washington or is agreed at the Nato summit this week in Wales.

But he signalled some flexibility, when, Cameron said in reference to air strikes that "we should ask ourselves how we best help those people on the ground who are doing vital work in countering [Islamic State]".

He added that, as in Libya, "the British government must reserve the right to act immediately and inform the House of Commons afterwards".

Cameron was even unable to win unambiguous backing from his Lib Dem partners for plans to give police powers to force suspected terrorists to relocate within the UK, amid criticism they amounted to a watered-down version of the controversial control orders introduced by Labour but dropped by the coalition.

Lib Dem sources said they had not definitely signed up to the plans for relocation, but said they felt a duty to look at the proposal after it had been recently recommended by the government independent reviewer of terrorism, David Anderson.

Cameron said he would go ahead with plans to give UK police the power to revoke a passport from a UK citizen – a power currently confined to the home secretary under the royal prerogative.

However, the power to revoke will last 30 days and be subject to judicial review. "It will not be possible for a UK policemen to withdraw a passport at the border on a whim," a Lib Dem source stressed.

All international airlines operating in the UK will also be required by statute to hand over information on passengers travelling from the UK.

The American-style powers are seen as necessary as some airlines have been failing to provide information early enough for British intelligence agencies to prevent a passenger from travelling.

Cameron added that the government's Challenge programme would be put on a statutory footing so anyone suspected of being radicalised will be required to go on a government re-education programme, removing any voluntary element.

Overall, the package announced to MPs by Cameron appeared at best incomplete and certainly less dramatic than suggested when he called a rare Downing Street press conference on Friday to announce the terror threat was being raised from substantial to severe for the first time in three years.

He also rejected proposals from the mayor of London, Boris Johnson, to criminalise travel to certain individual countries or to change the criminal standard of proof.

He said: "The government are clear that it would be wrong to deal with the gap by fundamentally changing core principles of our criminal justice system."

But there were also no signs of a coalition split on the issue, with neither Liberal Democrat nor Conservative sources eager to turn any disagreements over civil liberties into a political clash, aware good will is needed to reach agreement on what they both regard as a new national security threat.