North Korea has banned Malaysian citizens from leaving the country, and Malaysia has responded with a tit-for-tat ban in an escalation of the row over the killing of the North Korean leader Kim Jong-un’s half-brother in Kuala Lumpur.

The state news agency in North Korea cited a foreign ministry statement saying all Malaysians “will be temporarily prohibited from leaving the country until the incident that happened in Malaysia is properly solved”.

Malaysia’s prime minister, Najib Razak, called it an “abhorrent act, effectively holding our citizens hostage” in disregard of international law. He said he had instructed police to prevent all North Korean citizens in Malaysia from leaving until the safety of Malaysians in North Korea was assured.

Malaysia’s foreign ministry said it had 11 citizens in North Korea, including three embassy staff, six family members and two others. Neither country has announced how many North Koreans live in Malaysia.

A Malaysian police investigation into the murder of Kim Jong-nam at Kuala Lumpur airport last month has infuriated Pyongyang and named seven North Koreans wanted for questioning by police.

Three North Korean nationals wanted in connection with the killing are hiding in the country’s embassy, Malaysia’s chief of police said on Tuesday.

An employee for North Korea’s state airline named as a suspect and a senior diplomat called in for questioning are believed to be inside the embassy.

A photo made available by the state news agency of North Korea shows a ballistic rocket launch. Photograph: KCNA/EPA

“We are trying to physically identify all the embassy staff who are here,” the deputy Malaysian home minister, Nur Jazlan Mohamed, told reporters outside the North Korean embassy. He added that staff would not be allowed to leave the building “until we are satisfied of their numbers and where they are”.

In the past three days, North Korea and Malaysia have declared each other’s ambassadors as “persona non grata”, ordering them to leave the country.

North Korea’s ambassador, Kang Chol, had attempted to block the investigation and prevent an autopsy on Kim Jong-nam’s body. Lab test results later found Kim was killed with the nerve agent VX, a banned chemical weapon known to be produced in state laboratories.

The ban on North Koreans leaving Malaysia will largely affect a community of expatriates known to live in Kuala Lumpur, some of whom run restaurants. There are also North Korean labourers working for Malaysian mining companies.

The Malaysia Star newspaper quoted an unnamed Malaysian official as saying diplomatic staff in Pyongyang were safe. “We have communicated with our staff and are in touch with them. They have been advised to lead their lives normally,” the official said.

Two women – one from Vietnam and another from Indonesia – are believed by Malaysian police to be the two assailants captured on CCTV cameras grabbing Kim Jong-nam’s face. Both were arrested and have been charged with murder.

They face the death sentence if convicted, but police reports suggest they did not mastermind the murder. According to Jakarta’s deputy ambassador to Malaysia, Indonesian national Siti Aisyah was paid $90 (£72) for what she believed was a prank.

Pyongyang has not identified the deceased man as Kim Jong-nam and says he died from a heart attack. Malaysian police have called for next of kin to come forward to identify the body and provide DNA samples.

Friends of Kim Jong-nam have told the Guardian they believe his family has gone into hiding following the attack.

In the first report from North Korea’s KCNA news agency since the attack, the government accused Malaysia of breaking international law by conducting autopsies on a diplomatic passport holder and withholding the body.

“This proves that the Malaysian side is going to politicise the transfer of the body in utter disregard of international law and morality and thus attain a sinister purpose,” it said.

Last month, North Korea lashed out at Malaysia, accusing it of having a “sinister purpose” and collaborating with South Korea, which has said Pyongyang agents assassinated Kim Jong-nam.

Reuters contributed to this report