Muslim protesters infuriated by cartoons depicting the prophet Muhammad raised the diplomatic stakes last night as Iran's best-selling newspaper announced it would retaliate by running images satirising the Holocaust.

The decision by the rightwing Hamshari daily to launch an international competition to find the most suitable caricatures came as demonstrators hurled firebombs and stones at the Danish embassy in Tehran and the Iranian government imposed a formal trade ban on Danish imports. Last night mobs were attempting to storm the Danish compound.

In London, the home secretary, Charles Clarke, branded the activities of Islamist protesters outside the Danish embassy last week as "unacceptable". He told MPs he was pleased the response to the publication of the Danish cartoons had "in general been respectful and restrained in the best traditions of British tolerance". But he added: "If the police conclude there have been breaches of the law and decide to take any action, we would, of course, support them."

Scotland Yard has set up a special squad to investigate the demonstrations and the placards on display which called for the death of those who "insult Islam". The inquiry will examine everything from video recordings made by officers to photographs published in newspapers.

One protester, who was photographed apparently dressed like a suicide bomber, apologised "wholeheartedly" yesterday. Omar Khayam, 22, appeared outside his house in Bedford accompanied by the chairman of his local mosque and his local MP. He said he had not intended to cause offence to the victims of the July 7 London bombings or their relatives but added that his protest remained valid because of the hurt caused to Muslims by the publication of the Danish cartoons.

"I felt the Danish newspaper had been provocative and controversial, deeply offensive and insensitive. But by me dressing the way I did, I did just that, exactly the same as the Danish newspaper, if not worse. My method of protest has offended many people, especially the families of the victims of the July bombings. This was not my intention."

Fresh protests raged across the Muslim world throughout the day, claiming four lives in Afghanistan and one in Somalia, but the involvement of the Iranian authorities added a further twist to the diplomatic tensions between western governments and Tehran.

Hamshari is owned by Tehran city council and its plan follows a string of anti-Zionist statements by Iran's hardline president, Mahmoud Ahmadinejad, who has dismissed the killing of 6 million Jews by the Nazis in the second world war as a "myth" and called for Israel to be "wiped off the map".

Farid Mortazavi, the paper's graphics editor, said the cartoons would be published to test the argument of western newspapers which have cited freedom of expression in printing the prophet Muhammad images.

"The western papers printed these sacrilegious cartoons on the pretext of freedom of expression, so let's see if they mean what they say and also print these Holocaust cartoons," Mr Mortazavi said.

There were attacks earlier in the day on the Austrian embassy in Tehran. Iran's trade with Denmark is worth around £160m a year.

In Afghanistan, police opened fire on 2,000 protesters as they tried to break into the main US base in Bagram, north of Kabul, killing two and wounding five, said local chief Kabir Ahmed. No US soldiers were involved. Another two demonstrators died in the central town of Mehtarlam, after police fired live rounds into a crowd of demonstrators burning tyres and hurling rocks. A shot from within the crowd provoked the police into shooting, officials said.

An Austrian newspaper, Kleine Zeitung, became the latest European publication to print the controversial cartoons. The paper admitted several of its Muslim distributors had refused to sell the day's issue and said some had torn the offending caricatures out.

Several European countries, including Austria, have now warned citizens not to travel to the Middle East. The cartoons have been published in virtually every major country across Europe - except Britain. Ukrainian papers published the cartoons yesterday, joining Denmark, Norway, France, Ireland, Germany, Italy, Spain, Switzerland, Bulgaria, Hungary, and Poland.

Speaking from Beirut, Omar Bakri Mohammad, the leader of the Islamist group al-Muhajiroun which is banned in Britain, called for those who "blasphemed" against the prophet to be executed.

"In Islam, God said, and the messenger Muhammad said, whoever insults a prophet, he must be punished and executed," he told BBC radio.

Political and religious leaders in Lebanon attempted to calm sectarian tensions a day after Islamic extremists torched the building housing the Danish embassy and rampaged through a Christian quarter of Beirut. Lebanon's interior minister, Hassan Sabei, who resigned on Sunday, blamed a hardcore of infiltrators.

The protests spread even further across the Muslim world. There were fresh disturbances in India, Indonesia, Palestine and Thailand. In Somalia, a 14-year-old boy died during clashes with police in the port city of Bosasso after protesters hurled stones at offices used by international aid agencies.