In 1924, New York publisher Condé Nast built a duplex apartment on a rooftop on 1040 Park Avenue. Decorated by interior designer and socialite Elsie de Wolfe, the grand space, with its 75ft conservatory and multiple entertaining spaces, became famous for lavish parties. It was one of the earliest examples of the penthouse apartment – a typology that has since been associated with luxury, where those with means can enjoy a home and open space in the heart of the city, far from the crowded streets below.

A few decades later, a different style of rooftop abode emerged in the capital of the newly independent India. Influenced by the Garden City movement and the legacy of architect Edwin Lutyens, who designed much of Delhi, post-independence planning rules aspired to create a low-rise city with plenty of breathing space.

Buildings in Delhi’s residential areas were restricted to two storeys, with construction permitted on only a fraction of the space on the third floor, so on top of homes, families built small dwellings for their own use, as accommodation for domestic staff or to rent out cheaply. Exposed to the elements, the single room on the top floor became known evocatively as the barsati – derived from the Hindi word for rain, barsaat. These apartments – generally, a small shack with a large terrace – afforded a new generation of urbanites cheap living space near the centre of town.

The rooftop space gives a sense of intimacy with your neighbours, in an otherwise rather brash city

But it’s a typology that, as land values rise and the population grows, is fast disappearing. While there are no official figures, anecdotal evidence suggests that in the 1980s, 75% of small rentable properties in many residential suburbs were barsatis – the same areas today have only a few dozen such properties, but many more apartment blocks. “This city has transformed dramatically,” says Rahul Rewal of Delhi-based estate agent Mark Estates. “The barsati is now a thing of the past.”

The result is a city that is increasingly dense, but also increasingly unaffordable for young workers, students and creatives. As the city’s boundaries move further outwards, those who might have once lived close to the centre now occupy flats in gated condominiums on the city’s periphery. The underlying story of urban sprawl and gentrification is replicated in cities across the developing world.

Rachit Tiwary’s barsati in the middle class suburb of Greater Kailash. Photograph: Debika Ray

The transgressive appeal of the rooftop space is not a recent development – in the 1920s, Mexico City’s rooftop azoteas became a hotbed of creative thinking. A movement is currently underway in Spain to claim rooftops for community use –privately owned spaces that are, because of their exposed position, very public. Events and performances taking place on these terraces are returning life to this once well-used level of these cities.

At the same time, rooftops spaces are being harnessed for their property potential. In Barcelona, city planning authorities have welcomed La Casa por el Tejado, a project to drop prefabricated homes on to the city’s abundant flat rooftops in an effort to increase the supply of homes. Under the scheme, the owners of the building give architect Joan Artes and his team permission to build an extra storey in exchange for carrying out improvements to the communal facilities. A more troubling example of capitalising on rooftops can be found in Hong Kong, where eye-watering property prices have resulted in a proliferation of rooftop slums: cramped, windowless shacks, often poorly constructed out of cheap materials, with facilities shared between several families.

High prices are gradually making barsatis an elitist concept, out of the reach of even middle-class city dwellers

Meanwhile, in Delhi the roofspace is vanishing. Since India’s economy liberalised in the 1990s, its planning laws have changed to allow increasingly high-rise construction in residential areas – raising the allowance to four storeys led to the demolition of family homes to make room for developer-built apartments. “Real estate in Delhi is so expensive now that it doesn’t make sense for anyone to not use buildable space,” says Sonali Rastogi, a member of the public conservation group Delhi Urban Arts Commission and co-founder of architecture practice Morphogenesis. “And that has lead to the decline of the barsati.”

While they were never really affordable for the city’s poorest, high prices are gradually making barsatis an elitist concept, out of the reach of even middle-class city dwellers, argues Rewal. “Many youngsters with their first jobs can no longer afford them so they are being transformed into boutique residences lapped up by expats from Europe.”

There was a time, he recalls, when light, open terraces were not a luxury and young working people and fledgling artists could afford to rent barsatis. As a result, the rooftops of Delhi become associated somewhat with the city’s artists and intellectuals. “Delhi may not have flourished in the manner of Mexico, where the azoteas were incandescent with creativity, but there is something about rooftop cities that encourages curiosity, and that mitigates against conservatism,” says Delhi-based journalist and author Nilanjana Roy. “Barsatis in south Delhi were affordable, and they encouraged a kind of open flow of ideas – one reason why artists and actors took to them.”

A staircase leads up to the upper terrace at Serena De Sanctis’s home in Bhogal. Photograph: Debika Ray

Writer and historian William Dalrymple lived in one too, when he first moved to Delhi, as did writer and activist Arundhati Roy, painter MF Husain and writer Pankaj Mishra. In Anita Desai’s short story, The Rooftop Dwellers, the protaganist Moyna moves into a simple rooftop room, whose “very bareness gave her the freedom to indulge her wildest dreams and fancies”. The terrace, where she entertains friends “just as she had imagined an adult working woman in the metropolis might do”, is a “clear, empty space under an empty sky, with a view of all the other rooftops stretching out on even side … giving her a sense of being the empress of all she surveyed.”

“The terrace was a lovely thing”, says journalist Ian Jack, who spent the 1970s living in a barsati in Defence Colony with his then-wife. “We erected a shamiana [tent] there when we married and I enjoyed the nightly sounds of the chowkidar [guard] and the freight trains hauled slowly by steam locomotives round the Delhi loop line. The rent would have been a few hundred rupees a month – the professional middle class then didn’t have much money. A lot of tea was drunk. Cars other than taxis were rare. The sun shone from blue skies all through the winter – amazing to think of now. That’s what I think of when I hear the word barsati.”

“Kite flying, sunbathing in winters, morning yoga,” recalls Rewal of the barsati’s heyday. “People used to store a charpai [bed] in the constructed area and sleep on the terraces during power outages, or just watch the rain fall on the exposed brick terraces. The bachelor or young children in the family ended up using the barsati for privacy.”

The view from Serena De Sanctis’ barsati in Bhogal, Delhi. Photograph: Debika Ray

The appeal of the rooftop space remains for some. Serena De Sanctis, an Italian photographer, has shared a barsati in a relatively affordable area, Bhogal, with her boyfriend since they arrived in Delhi in 2013. A narrow staircase by her front door leads up to an upper terrace, from which you can see far into the distance. A few metres away, on an adjacent rooftop, a woman is taking down clothes from a washing line. There is a sense of intimacy with your neighbours, in an otherwise rather brash city. “To open your door and have this view – it isn’t possible in a regular flat,” she says. It is a few days before Diwali and she expects the residents of the whole building to make use of the space. “Everyone comes up to use this terrace – especially during celebrations. Children fly kites.”

Increasing pollution and traffic in Delhi are starting to make such spaces unappealing. Writer and editor Dhritabrata Bhattacharjya Tato moved out of his barsati in the suburb of Jor Bagh four months ago, after a flyover and a tall office building were erected near his flat. “I have great memories of that place,” he recalls. “I had this huge terrace where I could have parties for 50-60 friends. My artist friends would display their work sometimes. But over the past three or four years, the area lost its charm.”

“Charm” is perhaps what is now lacking for many when it comes to the barsati. Rachit Tiwary had not heard the term before he moved into his apartment in the affluent suburb of Greater Kailash, walking distance from the television studios of news channel NDTV where he works. His spacious terrace has a small gazebo and a porch swing. But the cost of living in Delhi is become an increasing concern. “A quarter of my income goes on renting this place, which is crazy,” he explains. He hopes to move within the next six months to a two-bedroom apartment – most likely in the neighbouring city of Gurgaon, where properties are more affordable, and modern – offering the convenience of communal facilities and maintenance.

Many young professionals now live on the city’s periphery, in “satellite cities” such as Gurgaon and Noida, vast new cities of gated developments, malls, offices and highways. Economic factors have made the shift inevitable, but has such spatial dispersion been accompanied by a decline in creativity and social radicalism? Roy appreciates the concern. “Up to the early 2000s, the city’s architecture was markedly friendlier and more open than it is now,” she says. “The gated communities and walled enclaves that have come up are more formal, perhaps more intellectually conservative, spaces. The sense that a lot of Delhi’s intellectual life now takes place in small, gated communities of the mind is hard to ignore.”

But she cautions against overly romanticising Delhi’s previous form. “Friends in Gurgaon say that a thriving intellectual community is gradually developing there.” Rastogi agrees: “The hubs around Delhi are now maturing to a level of cultural independence – there are events, theatre, art, and independent social activism. After all, how long can such a rapidly sprawling city continue to believe that its centre will always form every aspect of its inhabitants’ cultural experience?”

Follow Guardian Cities on Twitter and Facebook to join the discussion