Two elderly passengers taken off the Diamond Princess cruise ship because they were infected with a new virus have died, Japan’s health ministry said Thursday, becoming the first fatalities from the virus-stricken vessel.

Japan now has three deaths linked to the COVID-19 illness.

Japan’s NHK public television said both were Japanese in their 80s. A health ministry official only confirmed that they had been previously been hospitalized in serious condition and had existing chronic diseases. The official spoke anonymously, citing office protocol.

The new virus began in China late last year has sickened tens of thousands of people, mostly in central China’s Hubei province. The 621 cases confirmed among the Diamond Princess’s original 3,711 people on board are the most anywhere outside China.

The Diamond Princess, docked in Yokohama port, near Tokyo, started letting passengers who tested negative for the virus leave the ship Wednesday. Test results are still pending for some people on board.

Japan’s government has been questioned over its decision to keep people on the ship, which some experts have called a perfect virus incubator.

Health Minister Katsunobu Kato initially said those with negative virus tests had fulfilled the Japanese quarantine requirement and were free to walk out and go home on public transportation. Later Wednesday, he urged the former passengers to refrain from non-essential outings and try to stay home for about two weeks.

“COVID-19 is not 100% known, and a lot of people got infected on the Diamond Princess. Taking those factors into consideration, we believe taking extra caution will contribute to preventing the risk of future infections,” he said.

About 500 passengers had left the ship by Wednesday evening, and Japanese officials were to spend the next three days disembarking about 2,000 others. The Diamond Princess was quarantined after one passenger who left the ship earlier in Hong Kong was found to have the virus.

Crew members, who couldn’t be confined to their rooms because they were working, are expected to stay on the ship.

Japan’s Chief Cabinet Secretary Yoshihide Suga said a more controlled health watch for the crew would start immediately because they can isolate themselves by spreading out and using vacated passenger rooms.

Before the quarantine on the ship had ended, the United States evacuated more than 300 Americans and put them in quarantine in the U.S. for another 14 days. South Korea, Australia and Hong Kong evacuated their residents for quarantines as well, and Canada and Italy sent flights for their citizens as well.