Britain last night fired the first shot in a potentially explosive diplomatic row with Israel by calling in the country's ambassador to explain the use of faked British passports by a hit squad who targeted a Hamas official in Dubai.

The Israeli ambassador has been summoned to the Foreign Office to "share information" about the assassins' use of identities stolen from six British citizens living in Israel, as part of the meticulously orchestrated assassination of Mahmoud al-Mabhouh.

Britain has stopped short of accusing Israel of involvement, but to signal its displeasure, the Foreign Office ignored an Israeli plea to keep the summons secret. "Relations were in the freezer before this. They are in the deep freeze now," an official told the Guardian.

Gordon Brown yesterday launched an investigation into the use of the fake passports, which will be led by the Serious Organised Crime Agency (Soca). The British embassy in Tel Aviv is also contacting the British nationals affected in the plot, "and stands ready to provide them with the support that they need", the Foreign Office said in a statement last night.

"The British passport is an important part of being British and we have to make sure everything is done to protect it," Brown told LBC Radio yesterday.

Israel's foreign minister, Avigdor Lieberman, insisted there was no proof that the Mossad was involved in Mabhouh's killing in a Dubai hotel last month, but added that Israel had a "policy of ambiguity" on intelligence matters.

However, there were calls in Israel for an internal government inquiry into whether the Mossad was responsible for identity theft from dual nationals, and criticism of its chief, Meir Dagan, for what critics described as a clumsy operation that risked alienating European allies.

"What began as a heart attack turned out to be an assassination, which led to a probe, which turned into the current passport affair," a columnist, Yoav Limor, wrote in Israel Hayom, a pro-government newspaper. "It is doubtful whether this is the end of the affair."

Israel's ambassador to London, Ron Prosor, will meet Peter Ricketts, head of the diplomatic service and the permanent under-secretary at the Foreign Office.

Yesterday more details emerged about the assassination plot:

• The Guardian learned that a key Hamas security official is under arrest in Syria on suspicion of having helped the assassins identify Mabhouh as their target.

• Reports that the hit squad could have been bigger than the 11 suspects named by the Dubai police appear to have been confirmed by surveillance pictures showing other possible accomplices, including a second woman.

• Authorities in Vienna have begun an investigation into whether Austria was used as a logistical hub for the operation, after seven of the mobile phones used by the killers had Austrian SIM cards.

• Three of the killers entered Dubai with forged Irish passports that had numbers lifted from legitimate travel documents.

It is not the first British-Israeli row over the misuse of British passports. British officials are particularly angry because the Israeli government pledged that there would be no repeat of an incident in 1987, in which Mossad agents acquired and tampered with British passports.

Lieberman said he believed that relations with Britain would not be damaged. "I think Britain recognises that Israel is a responsible country and that our security activity is conducted according to very clear, cautious and responsible rules of the game. Therefore we have no cause for concern," he said.

However, the former Liberal Democrat leader Sir Menzies Campbell, a member of the Commons foreign affairs committee, welcomed the decision to confront the Israeli government directly. He said: "If the Israeli government was party to behaviour of this kind it would be a serious violation of trust between nations."

France yesterday also claimed that the French passport used by one of the assassins had been forged. A source close to the French intelligence services told Reuters a French passport which Dubai said had been used in the operation had a valid number but incorrect name. "It was a very good fake," the source said.

Hamas, meanwhile, vowed vengeance for Mabhouh's assassination. At a memorial rally in Gaza, Hamas militants vowed that the movement's armed wing, Izz-el Deen al-Qassam, "will never rest until they reach his killers".