It was depicted as a haven for drug lords and assassins in the TV series Homeland, lauded as an experiment in social empowerment at the Venice Architecture Biennale and featured in countless articles and documentaries.

For the past eight years, the Tower of David – a half-built skyscraper in downtown Caracas – has been home to thousands of squatters who transformed the abandoned block into a vertical slum complete with grocery shops, tattoo parlours, internet cafes and a hair salon.

This week, however, looks like the beginning of the end for the ramshackle community, as city authorities started to move the first of the tower's inhabitants to a new social housing complex in Ciudad Zamora, more than an hour's drive from the Venezuelan capital.

The relocation comes after three months of negotiations between government officials and representatives from the tower, and will entail moving the inhabitants out three floors at a time until the 27 inhabited storeys of the 52-storey skyscraper are emptied out.

Residents with their belongings. Photograph: Federico Parra/AFP/Getty Images

Residents of the tower had long known they would one day have to leave, but many were still sad to go. "We were aware we had no rights over our homes here so we knew this day would come," said Alfonso Garcia, who moved into the tower with his wife and two young daughters five years ago. "Like all changes, it is a bit scary but we are all happy to be finally given a house," Garcia said.

The mood was solemn as the first group of about 300 residents waited on Tuesday for the buses that would take them to their new homes. Dressed in fluorescent yellow reflective vests, and clutching pets and sportsbags, they were watched by riot police and troops. Some expressed a thinly veiled sense of defeat."They didn't force us out, so that's good," said Yecenia Polanco, who is due to move with her three boys in the second round of relocations. "But they could have also given us houses closer to where we are now and to where our children go to school."

Ernesto Villegas, the minister for the revolutionary transformation of greater Caracas, said all the tower's residents would be relocated in "dignified homes", and promised that a new school would be opened in offices at their new housing complex. "This is not an eviction, but rather a relocation," he told reporters. Villegas said several children had fallen to their deaths from the tower, which in some places is lacking walls or windows.

Inside the Tower of David. Photograph: Federico Parra/AFP/Getty Images

What happens to the tower itself remains unclear. Villegas denied local media reports that a Chinese consortium had bought the building to use as its headquarters and said the building's infrastructure must be evaluated before a decision could be made over its eventual use.

Built in the mid-1990s by the Venezuelan tycoon David Brillembourg to serve as the headquarters of his bank, the high-rise was left unfinished after a banking crisis left the country's economy in tatters. The tower was first occupied by squatters in 2007, and eventually became home to more than 1,200 families.

Its symbolism has shifted with the years. Originally meant to stand as an emblem of the country's vast oil wealth, the building was seen by some as a testament to a failed capitalist model; others saw the towering squat as a symbol of the many shortcomings of the late Hugo Chávez's Bolivarian revolution. "This is an ongoing process. It is still too early to say what this new move will be emblematic of," said Villegas.

Residents walk to a bus which will take them to their new homes. Photograph: Jorge Silva/Reuters

What began as a tent camp grew over the years into a thriving if highly regulated community under the rigid management of a born-again ex-convict who had served time in prison, Alexander "el Niño" Daza.

Inhabitants in the surrounding neighbourhoods complained that the tower provided a safe haven for gangsters and other criminals, and at least one high-profile kidnap victim was rumoured to have been held in the building. Daza was incarcerated in November, and told the Guardian last December that he believed the government was trying to undermine his leadership in the tower.

Among those watching the first group of residents leave on Tuesday, several expressed anger at being forced from their homes. Miriam Figueroa, who runs a shop from her flat, said she still didn't know where her new home would be. "I've been here since the beginning. I carried every single one of these cinder blocks on my back up all those flights of steps. The government isn't offering to recognise that effort, or even the cost of all of these materials. If I refuse the one option they've offered, I am back in the street."