Back in the 1980’s, when Louis Smuts was growing up in Johannesburg, South Africa, his family couldn’t go outside together without risking arrest. “My mother would always walk behind [my father] wherever they went,” he recalls. And in the car, she would sit in the back and pretend to be the family maid. At the time, Smuts didn’t understand that only white people could move freely in the city, while black South Africans were pushed to the outskirts. The country’s white-ruled government called this system by the Afrikaans word “apartheid,” meaning “separateness.” Under apartheid, white people had access to the best schools, jobs and healthcare.

Smuts’ father was white and his mother was colored—a South African term for people of mixed race. They had gotten married in neighboring Swaziland, but back home their marriage was illegal.

Apartheid leaders claimed that segregated cities were better for everyone, and apartheid was strictly enforced. Police would patrol neighborhoods to make sure that white people and black people weren’t living together. But the system was never airtight, and people found creative ways to slip through the cracks.

“If you lived in a high rise building with five or six hundred flats in it,” explains Smuts, “they weren’t going to go through each and every flat to come look for any illegal persons.” And over the years, many interracial families like the Smuts disappeared into one high rise apartment building in particular—a tower called Ponte City. “I don’t think there’s anybody in Johannesburg that doesn’t know Ponte,” Smuts says. “They might not have been there. But everybody knows Ponte.”

Looking at Johannesburg’s skyline, the 54-story tower is hard to miss—it is the tallest apartment building on the African continent. It is also distinctively shaped, a massive cylinder with an empty central core. The billboard wrapping its top like a crown also helps it stand out. For many, the building symbolizes Johannesburg—because over the past four decades, its fortunes have reflected the changing city around it.

“Because of its scale, size, its checkered history, its toughness, and roughness, [Ponte speaks] to a very Johannesburg condition,” says architectural historian Melinda Silverman. “It has spoken to a city which goes very rapidly through cycles of decline and prosperity and decline and prosperity.” Ponte has always been a kind of vertical waiting room for admission to South African society. But it’s also been a laboratory, a place where the city seems to try out new versions of itself.

Before Ponte became a hideaway for interracial families like the Smuts, the building attracted a much different kind of clientele.

At the start of the 1960’s, the apartheid system had been firmly in place for more than ten years and South Africa’s economy was a rising star in the world. It had just given birth to a new currency, the Rand, which was already stronger than the US Dollar. Its success was largely driven by its access to cheap black labor and the high value of gold.

Over the next decade, foreign investment in South Africa doubled, but skilled workers were in short supply. Because the government had denied the majority of the black population a decent education, it had to fill the gap by recruiting single white men from all over Europe. The country’s white population increased by over 50% between 1963 and 1972. They were attracted in part, explains Silverman, by “a level of privilege that you probably couldn’t have found in any other part of the world. If you’re a middle class person, living in England or America, you probably had to wash your own dishes, and make your own beds, whereas if you were a middle class person living in South Africa, you probably had at least two people doing that for you.”

One particular square mile of central Johannesburg—the future site of Ponte City—served as a regular landing place for incoming European migrants. It was home to bars and live music joints, a place where specialty stores hawked French magazines, Italian shoes and American rock-and-roll. It had a shiny, vibrant, bohemian edge that people compared to New York’s Greenwich Village and London’s Soho District.

At the same time, it felt almost like a foreign country. As with most of Johannesburg’s nicest neighborhoods, this area was zoned exclusively for white people. Black South Africans could work there, but unless they were live-in servants they had to be out by nightfall each evening.

Developers could hardly satisfy the demand for high rise apartments in this part of the city. Buildings kept getting bigger and taller, monuments to a thriving economy. “This was white people wanting to make their mark on the landscape,” says Silverman, to show that “South Africa is a modern, progressive, impressive place.”

Of course, South Africa was neither truly modern nor progressive at this time. Across the continent, change was underway; this was the era of African independence. Old colonies were falling like dominoes and by the mid 1960s, most African countries were black-ruled. To much of the world, South Africa’s white government began to look pretty backward. And it didn’t help that the country had recently thrown several of the anti-apartheid movement’s most powerful leaders in jail for life, including a young lawyer named Nelson Mandela.

Nonetheless, Johannesburg kept developing, and in the middle of it all rose Ponte, a great round tower completed in 1975. Rodney Grospkoff, one of three architects who designed the building, describes the structure: “The whole outside of it is made out of raw concrete. And then at its foot it’s really anchored,” he explains, describing the big base that spreads out at the bottom. “It’s so beautifully done that people once upon a time said it looks like a wedding cake.”

But not everyone shared this affection for the massive, brutalist structure. “Straight after it was built, it was voted as the second ugliest building in Johannesburg.” Second ugliest or not, Ponte City was popular. Residents started moving in even before the building was finished, lured by its furnished flats and panoramic city views. All the units in Ponte faced both inward and outward, with entrances wrapping around the central core and views out the sides.

Ponte was designed to be like a city within a city, featuring housing above but also shops below. There was even talk of putting a miniature ski slope inside its hollowed-out core.

But even as Ponte’s new residents were moving in, the city around them was cracking apart. At the time it was built, Johannesburg’s inner-city was almost exclusively white. Black South Africans lived on the edges of the city, in communities called townships. And about twelve miles southwest of Ponte, in a black township called Soweto, a new revolt against apartheid had begun.

Protests escalated and on June 16, 1976, police murdered at least 176 high school students during a peaceful protest march. Images of the massacre quickly circled the globe, inspiring renewed calls to end white rule. The Soweto uprising also sent Johannesburg’s urban planning dreams crashing down. International sanctions and boycotts in the wake of the violence helped tank the country’s economy. It became difficult to keep all the newly constructed high-rises fully occupied.

Meanwhile, like many cities around the world, Johannesburg had also started to experience a wave of suburbanization, with white people moving out of the inner city and into new suburbs. And for black South Africans fleeing from the poverty and violence of the townships, the inner city was an alluring option, especially Ponte.

It was still illegal for black, colored, and Indian South Africans to live in these areas, but the scale and density of the inner city made the laws harder to enforce. A new term was coined to describe the increased mixing going out on—”graying“—and for many, gray areas were refuges, impossible for the police to close down and search all at once. Landlords would turn a blind eye to skin color, but, in exchange, illegal residents had to deal with rent increases and poor maintenance.

projet: aller Au Bal là où M. Subotsky expose les 54 étages de Ponte City… pic.twitter.com/XrGORTkRnG — boussarie aude (@BoussarieAude) January 31, 2014

Inside Ponte, apartments started to get crowded and grimy. Plumbing broke down, and trash began to fill the open core in the center of the building like an oversized concrete garbage can.

Pretty soon, banks stopped issuing home loans in the area around Ponte—known as Hillbrow—as white people kept moving out. Between 1983 and 1993, Johannesburg’s inner city went from 20 percent black to 85 percent. So by the late 1980s, “white suburbanites would have been terrified of Hillbrow,” explains Silverman, but for black and colored people: “Hillbrow would have been this great beacon of opportunity,” a way into urban life.

Then, on February 2nd, 1990, President F.W. De Klerk went in front of Parliament to give his annual State of the Nation speech. Much to the shock of the audience, he announced the end of a long-time ban on many anti-apartheid political parties and liberation movements. He also announced that, after 27 years, the government planned to release Nelson Mandela unconditionally. From all over the world, exiled political activists began to return, eager to make their mark in a dramatically changing country.

Anti-apartheid activists who had fled South Africa, like Lentswe Mogatle, returned and found homes in Ponte. “It was like being in New York,” he recalls. Many of the old European cafes from the 1950s and 60s were still standing, serving late-night espressos and thick schnitzels. But alongside them were new shops and nightclubs, including one run by the famous South African trumpeter Hugh Masekela, who had also recently returned from his own exile.

Thabo Mbeki, who would later become president of South Africa, lived in the building next to Ponte. At the time of the country’s first democratic election in 1994, one observer counted 50 of the new members of parliament staying in the area.

But the end of apartheid didn’t represent the end of Ponte City’s evolution. Among other things, the transition to black rule in South Africa also meant the end of the country’s tightly closed borders, which for decades had cracked open only for white immigrants and a few laborers from the surrounding countries.

Throughout the 1990s, immigrants began arriving in Johannesburg by the tens of thousands. Many came from nearby Congo, Zimbabwe, or Malawi. Others traveled from as far as Nigeria or Pakistan. Some fled persecution. And many of them landed in Ponte Tower.

But the rush of immigration didn’t help the neighborhood’s physical decline. Like the residents of the 1980s, many new migrants had few legal protections against unscrupulous landlords. Ponte, the giant concrete trash bin at the center of it all, became the ultimate symbol of the neighborhood’s decline. There was no clearer, more distressing sign of how far things had fallen than when an American architect announced in 1998 that he was interested in turning the tower into a jail. In the end, Ponte never became a prison, but the symbolism was clear: not many people wanted to live in this building by choice.

Many of Ponte’s flats were eventually abandoned, while others were crammed with people trying to scrape together the monthly rent. The building, and the neighborhood, experienced a wave of violence and prostitution. Criminals took over the abandoned high rise and ran it like a vertical slum.

Developers made a few attempts to gentrify the tower and to turn it into luxury apartments again, but ultimately those plans didn’t really work. Ponte remains a lot like it was at the end of apartheid — a home for recently arrived immigrants from all over Africa.

Like South Africa itself, Ponte has gone from a symbol of white opulence to something far more complicated. It’s hopeful and it’s a little a rough around the edges. It’s a microcosm of the country’s history, but it’s also a place that moves on. And the strange concrete tube at the center of Johannesburg’s skyline continues to play the same role for newcomers that it always has. “The diversity is the same,” says Silverman. “It might have been Czechs, Portuguese, Italians and Danes living in Hillbrow then, but now it’s Cameroonians, Angolans, Nigerians and Zimbabweans. It always has been your entry point into the city.”