Almost 100 people are presumed dead after a landslide caused by Typhoon Mangkhut enveloped a small mining town in the Philippines, burying homes and a chapel where dozens of people had taken shelter.

By Monday night 36 bodies had been recovered in the remote town of Itogon, with 12 pulled from the ruins of the chapel. The area’s death toll was expected to rise with more than 50 people missing.

Mangkhut, a category five typhoon, swept through the Philippine region of Luzon on Saturday, wreaking destruction on homes and crops and causing massive flooding.

At the height of the storm, dozens of people in Itogon – mostly miners and their families – took refuge in a chapel housed in a former bunkhouse in the belief they would be protected. However, part of a mountain collapsed on top of the building.

Relatives of those lost gathered at the site in the aftermath but by Monday evening hopes of finding any survivors were diminishing as rescuers paused their efforts.

“I am 99% sure the people there are dead,” said the mayor of Itogon, Victorio Palangdan. “We will continue until we get them all.”

Typhoon Mangkhut – in pictures Read more

During the day about 300 rescuers used shovels and their bare hands to claw through mounds of rocky soil. A human chain was formed to pass rocks, debris and tree trunks out of the search area.

The landslide left a gaping gash in a hillside studded with small homes topped with rusting metals roofs. With damage to roads preventing the entry of heavy equipment, soldiers, police and miners used shovels to channel water from a nearby stream to loosen the earth.

It was excruciatingly slow work, as anguished relatives watched and waited for word on their missing loved ones. Recovered bodies were draped in fabric and lined up in a row at a makeshift tent on a road above the chapel.

Jonalyn Felipe, whose husband, Dennis, a small-scale goldminer in Itogon, was among those buried in the landslide, said she had tried to make him return to their home in northern Quirino province on Friday as the typhoon approached but he had refused.

“I was insisting because the storm was strong but he told me not to worry because he said they’re safe there,” she said, adding that her husband was last seen chatting with fellow miners in the chapel before it was hit.

Facebook Twitter Pinterest Rescuers use a line to transport a body recovered at the site where victims are believed to have been buried. Photograph: Aaron Favila/AP

Ricardo Jalad, the Philippines civil defence chief, said rescuers were racing to try to find survivors in the mud. The military and police were being supported by rescue teams, engineers and geologists, he said.

The environment secretary, Roy Cimatu, flew to the area on Monday to investigate what had happened. “We will not stop until we will recover [people] – whether they are still alive – in the mining area in the place of that incident,” he said.

Palangdan said the police had tried to make the miners – who were working on the disused goldmine illegally – leave the bunkhouse chapel before the typhoon hit, but they had refused. “They thought they were really safe there,” the mayor said on Sunday.

Play Video 1:25 Violent winds, raging water, mudslides: Typhoon Mangkhut’s path of chaos – video

At least 64 people have been confirmed dead across the Philippines, and an estimated 5.7 million people in the country are said to have been affected by the storm. The livelihoods of thousands has been devastated as crops were flooded just a few weeks before the harvest.

The storm hit the densely populated southern coast of China on Sunday night. More than 2.4 million people were evacuated from the area, and four died in the storm, according to Chinese state media. The storm is now headed for Yunnan province, a popular tourist area.

The Hong Kong Observatory said Mangkhut was the most powerful cyclone to hit the city since 1979.

Agence France-Presse contributed to this report