Tainan, Taiwan (CNN) Time is running out to find survivors in the rubble left by a powerful earthquake that shook southern Taiwan.

But the process of holding people responsible for some of the devastation is just beginning.

Three people were arrested Tuesday, Xiao Bo-ren, the Tainan government's legal affairs secretary, told CNN. All are former executives of the real estate development company that built Tainan's 17-story Weiguan Golden Dragon tower, which collapsed when the magnitude-6.4 tremor hit early Saturday.

Lin Ming-hui, former chairman of the now defunct Weiguan company, and two other former executives -- Chang Kui-an and Cheng Chin-kui -- face charges of professional negligence resulting in death.

The ruins of the toppled high-rise are the most visible evidence of the massive quake, which killed at least 54 people in the normally quiet city of 1.9 million people, according to the Tainan City Disaster Response Center. Five days later, more than 100 people remained missing.

Photos: Magnitude-6.4 earthquake shocks Taiwan Rescue workers and excavation vehicles carry out their rescue and search operation on what was the top of the Weiguan apartment highrise which collapsed in the 6.4-magnitude earthquake in Taiwan on Friday, February 10, 2016. Hide Caption 1 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan A mother holds a picture of her son after a male body was recovered on February 10. Hide Caption 2 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan Blue cans appear to have been used in the construction of an apartment building on Saturday, February 6. Dozens were killed in the disaster and rescuers were still pulling people from the rubble days later. Hide Caption 3 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan Rescue personnel search for survivors at the site of a collapsed building, the Weiguan Jinlong residential complex, in Tainan on February 6. Hide Caption 4 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan A boy is rescued from a collapsed building in Tainan on February 6. More than 200 people were rescued from damaged structures, many from a 16-story residential building that collapsed in Tainan, officials told CNN. Hide Caption 5 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan Rescue workers search a collapsed building in Tainan. Hide Caption 6 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan Rescue workers search a collapsed building in Tainan. Hide Caption 7 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan Heavy smoke comes out from a collapsed building as rescue personnel search for survivors in Tainan. Hide Caption 8 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan A man breaks down after seeing the body of a relative recovered from a collapsed building in Tainan. Hide Caption 9 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan A rescue worker hangs from a crane while searching a collapsed building in Tainan. Hide Caption 10 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan A canine rescue team searches for the missing in Tainan. Hide Caption 11 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan Rescue workers remove a victim on a stretcher in Tainan. Hide Caption 12 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan A man is rescued from a collapsed building in Tainan. Hide Caption 13 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan Rescue workers search a collapsed building in Tainan. Hide Caption 14 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan Rescue personnel work at the site of a collapsed building in Tainan. Hide Caption 15 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan Rescue teams search the collapsed building in Tainan. Hide Caption 16 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan A woman is rescued in Tainan. Hide Caption 17 of 18 Photos: Magnitude-6.4 earthquake shocks Taiwan Li Bomin, an official from Tainan City Fire Department, told CNN that more than 100 rescuers were at the scene. Hide Caption 18 of 18

Soldiers and firefighters were using life detectors and other equipment to conduct night search and rescue operations Wednesday, Taiwan's defense ministry said.

Heavy machinery began drilling around the site Tuesday, starting a new phase of removing debris layer by layer, exposing a mass of tangled wires, broken glass and shards of cement. The hope is that this will let authorities get to any survivors more easily, though many onlookers feared it could kill them by causing concrete slabs to collapse.

"If there are still people breathing in there and they are harmed now, after surviving for days, that would be the greatest tragedy," said Mao Jiecheng, who had rushed from Taipei to volunteer after the quake.

Lin Jianfang is there, too, camping out around the clock with three generations of relatives. The rescue effort is heart-wrenchingly personal for them since Lin's brother, a 41-year-old engineer, was asleep in his apartment when the quake struck and hasn't been heard from since.

Clutching his wife's hand and coughing as pulverized building debris wafted through the air, Lin said, "We refuse to give up hope."

The search continues in the rubble this morning for 100+ still missing in Tainan after a deadly earthquake hit in the early dawn hours on Saturday. #taiwanearthquake @cnnscenes A video posted by Sophia Yan (@sophiacyan) on Feb 8, 2016 at 3:57pm PST

Restaurants, hotels, volunteers pitch in

They're hoping for a miracle like the discovery of an 8-year-old girl and her aunt almost 72 hours after they were buried, as reported by Taiwan's SETTV. Military doctor Li Biqing told reporters that the girl was in "normal condition" despite her ordeal.

Tainan's fire department said that about 103 people -- dead or alive -- were believed to be trapped in the Golden Dragon tower's ruins. Some are students who attended nearby Kun Shan University.

A woman named Li -- she wouldn't give her full name -- has been urgently waiting these last few days for news of her 20-year-old nephew.

"We absolutely will keep waiting -- we cannot give up hope, we won't let our hearts go," Li said tearfully. "We never thought something this terrible could happen to our beloved Tainan, and in the main city area -- what are the chances?"

JUST WATCHED Earthquake rocks city in southern Taiwan Replay More Videos ... MUST WATCH Earthquake rocks city in southern Taiwan 01:35

The Golden Dragon tower is located in Tainan's most densely populated neighborhood, according to city government statistics. That area has been both devastated and transformed by the collapse, not only by the swarms of rescue workers but by the businesses and residents who have converged to do what they can.

A pub, for instance, has been repurposed as an emergency response center. Volunteers have set up tents next door at a colorful temple -- a bright contrast to the grim, gray debris -- to distribute water, blankets, clothing and hot food for rescuers and relatives like Li and Lin. Restaurants have stayed open to offer food and shelter as well, while hotels are giving free accommodation to people who have lost their homes.

Emergency medical and military vehicles dot Yongda Road, where the building once stood, in parallel with bright paper lanterns hung up to celebrate the usually joyous Chinese New Year holiday.

It's the biggest annual celebration here, a time for family, friends, laughter and feasts, but there is no festive spirit in the air this year.

"We are too somber to celebrate the holiday," said Jerrie Wu, a volunteer who has been handing out snacks and hot drinks to those working on the scene.

"This is my home..how could I not help? Any little bit I can do to help, I will do" Chen Zhifeng, 23 quake volunteer pic.twitter.com/buIECAI2pm — Sophia Yan 嚴倩君 (@sophia_yan) February 9, 2016

'So many other buildings ... still standing'

Condolences have poured in from around the world, including from U.S. senators, Pope Francis and celebrities such as tennis great Venus Williams.

"We all join our Taiwanese-American community here in Los Angeles to mourn this tragic loss of life," Mayor Eric Garcetti said . "When we witness horrific incidents like this, we are reminded of the immense destruction a major earthquake can cause."

Amid the sadness, there's also anger and uncertainty about what happened with the Golden Dragon tower, especially given how most of the rest of Tainan escaped relatively unscathed.

Central News Agency reported that "jerry-building" -- cheap or flimsy construction -- "is believed to be one of the causes of the collapse."

Images that have surfaced of tin cans believed to be used in the construction of the tower have caused city residents to speculate about whether the building company cut corners when the high-rise went up.

"There are so many other older buildings in Tainan that are still standing. Why was it only this building that was completely destroyed?" asked Wang Xingyou, a city cab driver.

Shaking his head woefully, he sighed deeply.

"This is not a happy new year for Tainan."