SHIAO LIN, Taiwan (CNN) -- From the side of the road leading to the former village of Shiao Lin, a woman cried as she viewed the flattened plain of mud where homes once stood.

A 4-year-old boy, left, holds his mother's hand at a rescue center in southern Taiwan on Friday. more photos »

"All of my relatives -- including my older brother -- are gone," cried Luo Yong Ling. "Only my brother's son is alive. More than 10 of my relatives and just one left."

Local authorities estimate scores of people were buried when a massive mudslide caused by Typhoon Morakot covered the village. As many as 200 people from Shiao Lin could be trapped under five stories of mud, said Wang Yuchi, a spokesman for President Ma Ying Jeou.

Residents said more people than usual were in the village when the storm hit last weekend because Saturday was Taiwanese Father's day. On this day, many people return to their home villages to be with family.

As of late Friday night, the death toll from the typhoon was put at 121, but the office of Taiwan's president said the number of deaths could exceed 500. Fifty-three people were missing and 45 were injured, according to official numbers, government spokesman Andrew Cheng told CNN.

"We try to minimize the death tolls, but you know these mudslides are quite serious. We have done our best to do the rescue work," Cheng said.

Government authorities told CNN's John Vause Friday that they believe most of the people able to be rescued have been.

Rescuers tried to determine Friday whether the mud was stable enough to bring in excavators to begin searching for bodies in Shiao Lin.

Since the typhoon made landfall over the weekend, more than 31,000 people have been pulled from villages, according to official government figures.

Chen Chiu Lian, 76, who lives in Shiao Lin with her teenage grandson, Wang Hsin Hong, explained to CNN's Pauline Chiou how the typhoon hit.

"I had just finished eating. My grandson was taking a nap. It rained and rained. There was water to my left and to my right ... The next day, it was still raining.

"Our house was like a boat. The water was like an ocean. How can you escape? There was no way to escape. My grandson told me to swim. I was swimming and crawling through the water and debris." Watch her story of survival »

She said their home was the only one left in the village where about 160 once stood. The two eventually were rescued by a helicopter from the roof of the house, Chen said. They went to stay with friends.

In other remote mountainous areas of Kaohsiung County, where Shiao Lin is located, rescues also have proved difficult.

Torrential downpours, dense fog, rugged terrain and raging rivers have made it difficult to get through. Washed-out roads and bridges rendered ground rescue operations virtually impossible in the central and southern regions of the island, the National Disaster Prevention and Protection Commission said. Watch challenges facing rescuers »

Earlier in the week, the six-story Jinshuai Hotel in Jhihben collapsed into the Jhihben River after its foundations were eroded by surging floodwaters. Media reports showed the building toppling into the water with a huge splash. Those inside had already been evacuated.

Typhoon Morakot dropped 2.6 meters (102 inches) of rain on southern and central parts of this island this week, according to CNN International meteorologist Brandon Miller.

Rescue commander Lu Chin Hong said the wind and rain caused mudslides on each side of the valley where Shiao Lin sits.

Pan Yuan Yin, who also lost relatives to the mudslides, said he doesn't want to rebuild the village. Watch rescuers reach stranded villagers »

"I want this to become a memorial and let the dead rest in peace," he told CNN.

In the nearby village of Shing Ki, where a bridge collapsed, rescuers evacuated as many people as possible using a cable system strung over the river. However, when the cable slackened, workers got worried and stopped.

About 300 people left the village, but a dozen opted to stay behind, said Vause.

In some places, the mud is waist-deep, he said. The only way people can escape the village is to climb down the side of the valley and cross the river, which is swollen and fast-moving. It is fed by mud lakes at higher altitudes.

After hitting Taiwan, Morakot roared on to mainland China on Sunday, killing at least six people and displacing 1.4 million, authorities said.