To get a glimpse of what will eventually become the Lowline, a subterranean Eden being billed as the world’s first underground park, you have to swipe your MetroCard at the Lower East Side’s Delancey Street station, go down one flight of stairs, go down another, slither through a few characteristically congested subway corridors, and then up another flight, to the J train platform.

Here, in the crucible of Manhattan’s public transportation system, with its slow, industrial wheeze, is an abandoned space the size of a football field. Seventy years ago it was the Williamsburg Bridge trolley terminal, transporting city folk between boroughs. But since 1948 it’s existed in a state of dark, musty desertion, save for tall metal columns, a few men in hazmat suits and the outlines of the balloon loops in which the trolleys once turned, which will be integrated into the park’s walkways.

Lowline co-founders Dan Barasch and James Ramsey have imagined a future for this space, one in which city-dwellers take respite from the concrete jungle beneath it, thanks to remote skylight technology that filters sunlight underground through fibre optic helio tubes. The park, which they hope will provide one of the city’s most densely populated and aggressively gentrifying neighbourhoods with more green space, will have a ventilation system and a year-round garden. In 2016, shortly after a pop-up called the Lowline Lab demonstrated the project’s solar redistribution technology and drew 70,000 visitors, New York City finally greenlit the Lowline, which will cost about $80m to build.

How the Lowline will look when finished. Photograph: The Lowline

“People who are trying to innovate have to be a little crazy,” says Barasch, reflecting on the 10-year anniversary of the project’s conception over coffee at the nearby members-only Ludlow House, itself a monument to the neighbourhood’s rapid mega-development. “I just want to do something that has never been done before.”

Ramsey, an architectural designer, got word of the abandoned space in 2009, the same year the High Line – the sleek, elevated greenway-cum-tourist attraction built on what was once a New York Central Railroad spur – opened a couple of miles north. A friend introduced him to Barasch, who at the time was working in strategy and marketing for Google. “Very fast I drank the Kool-Aid and believed almost wholeheartedly that technology could solve all social problems,” he says of his time at the tech giant. “But I still didn’t feel like I was making the world a better place.”

That year, Ramsey – a Yale-educated former Nasa satellite engineer who in 2004 founded Raad design studio, which is spearheading the Lowline – brought Barasch and his entrepreneurial acumen into the fold. When they toured the space together, they saw in its dilapidated, old-world charm an opportunity for innovation, to merge their interests in technology, architecture and New York to make the most of the city’s cosmopolitan topography. The station tower, they observed, would make for a perfect kid-friendly treehouse.

What’s needed for communities to thrive are public spheres where you attract different kinds of people Dan Barasch

The initial name for the project was Delancey Underground, but the generally rapt reception to the High Line positioned Delancey as a kind of architectural antidote, both projects motivated by a desire to revive and repurpose run-down parts of the city’s transit system. Hence Lowline, which it’s been called ever since.

But for all its altitudinous beauty, the High Line hasn’t escaped criticism: in the eyes of many Chelsea residents, the park has turbocharged the gentrification process. And, despite nearly five million annual visitors, High Line co-founder Robert Hammond has rued the excess weight his team put on aesthetic considerations. “Instead of asking [public housing tenants] what the design should look like,” he told CityLab in 2017, “I wish we’d asked, ‘What can we do for you?’”

The Observer’s Design magazine, issue 1, April 2019.

Barasch, who is particularly intrigued by the reclamation of abandoned spaces and has just published a book on the subject, is hyperconscious of the paradox these endeavours pose. The book, Ruin and Redemption in Architecture, comprises dozens of architectural case studies, divided into Lost, Forgotten, Reimagined and Transformed: examples include Tate Modern, Berlin’s Tempelhof airport, Atlanta’s BeltLine and the Gemini Residence in Copenhagen. The book also features an introductory essay by Barasch, in which the fourth-generation New Yorker remembers being charmed by the old tenement buildings near his grandmother’s Lower East Side apartment. But despite his “undeniable excitement and sense of adventure” in “encountering an architectural ruin”, Barasch warns that “beautiful abandoned buildings can be transformed into far less inspiring structures than their original selves”. With the Lowline, scheduled to open in 2021, the pair hope to avoid such an outcome by blending the rough-and-tumble of old Manhattan with this new, technologically ambitious horticultural refuge.

“It’s easy to criticise things once they’re built: you can build a modern masterpiece and people will still bemoan what’s missing, the electrifying feeling of the abandoned space itself,” he says, noting that the viaduct where the High Line is was once a graffiti-strewn late-night destination for teenagers to take drugs and have sex. “Now we have this incredibly beautiful public space that’s a huge destination for New Yorkers but has also been referred to as a kind of corridor between glass towers, a hallway among mega-developments.”

Diagram of the Lowline.

Residents of the Lower East Side, a historic early 20th-century landing ground for European immigrants, are justifiably wary of their own neighbourhood coming under siege. Indeed, the process is already well under way: an area that was once the provenance of working-class populations and a certain grungy, New York flavour has become a hotspot for luxury residential developments such as the one being built next to one of the neighbourhood’s oldest and most quintessential attractions, Katz’s Deli.

Ramsey and Barasch, however, don’t see the Lowline as an addendum to the neighbourhood’s propulsive modernisation so much as a vehicle for community engagement. Barasch invokes the sociologist Eric Klinenberg, whose book Palaces For The People argues for a new “social infrastructure” or spaces, ranging from libraries and churches to community gardens and even sidewalks, that enhance civic life and human connection. The idea is that, with renewed investment in public and private institutions of this sort, these spaces function as a kind of social scaffolding. And Barasch insists that, as much as architectural dogma suggests otherwise, good design and social responsibility don’t have to be in contention with one another. In fact, he hopes his experience of “swimming between sectors” and Ramsey’s design bona fides will allow them to corral the funding, political support and communal goodwill to create something that’s both socially conscious and architecturally unprecedented.

“The Lower East Side has a very large proportion of public housing, many vulnerable residents, a very diverse population, and it’s developing rapidly,” Barasch says. “What’s needed for communities to thrive are public spheres where you attract different kinds of people. As populations grow and there’s that rising tension between affluent and low-income residents, the question becomes: what kind of equity are we providing these people when it comes to open space? What the Lowline would do is create that, and also create a year-round open space, which would be first of its kind.”

Solar technology on the roof of the Lowline.

On a practical level, it’s the prospect of green space in winter that appeals most to New Yorkers, who are constantly jostling for fresh air in a city clogged by high-rises and pure, molten commerce. Though Manhattan is peppered with hundreds of playgrounds and community gardens, they’re showing signs of age: on average they are 73 years old and were last renovated in 1997. Moreover, a 2018 report by the Center for an Urban Future estimated that the city would have to spend close to $6bn to bring its parks’ infrastructure up to speed. But with the city’s endorsement in 2016, the Lowline got a jolt of political goodwill, the ripple effect of which has been felt internationally; Ramsey and Barasch regularly entertain delegations from other cities, including the mayor of Paris, who spoke with them recently about her plans to build downward.

“A fresh assessment of the quality of life in our cities is long overdue,” Barasch says. “Against the backdrop of these broader social trends, abandoned spaces take on heightened importance. I feel disappointed learning of a beautiful abandoned site being turned into a luxury condo or new retail space. It feels like the lowest common denominator, what is most likely to happen in the absence of intervention, some kind of vision for these spaces. We can’t waste what we have.”

Follow Guardian Cities on Twitter, Facebook and Instagram to join the discussion, catch up on our best stories or sign up for our weekly newsletter