Look out across Atlanta from the top of the bald granite hump of Stone Mountain and the skyscrapers of the city’s three separate business districts – Downtown, Midtown and Buckhead – poke up through what looks like a carpet of treetops stretching to the horizon.

But beneath this green canopy lie hundreds of square miles of quintessential American suburbia, of strip malls and cul-de-sacs. With no mountains or coastline to limit the city during the postwar boom, it just kept growing. A few years ago, “the city in a forest” and its surrounds ranked as the most sprawling large metropolitan area in the country.

Many of the 5.5 million people in metropolitan Atlanta live in low-density single family houses “outside the perimeter”, the eight-lane concrete ring of Interstate 285. With few sidewalks and virtually no public transit, living in this vast sprawl means driving to work or school, driving to get dinner or meet friends, driving to shops and healthcare – if you can afford a car, that is. Another 2.5 million people are projected to move to the metropolitan area over the next 20 years.

The BeltLine is changing how people think about the city Ryan Gravel

Atlanta, though, has an opportunity to change. The BeltLine – a 22-mile ring of abandoned and active freight rail lines that is being slowly transformed into a transit and trails loop – offers the city a different possible future. Parts of it have already been turned into walking and cycling paths, with new restaurants, bars and homes popping up along it. Just last month, the Metropolitan Atlanta Rapid Transit Authority (MARTA) agreed $570m funding for light rail along 15 miles of the route.

The vast scale and coverage of the BeltLine – it takes in 45 neighborhoods, from some of the richest to some of the poorest – means it has the potential to shape the city in a far more profound way than similar schemes such as New York’s High Line. Get it right and a large part of Atlanta can look forward to a future of high-density housing and walkable neighbourhoods, of cycling and public transit – and other American cities could have a new 21st century model to follow.

But the chances of getting it wrong are high. People who have lived in poorer, traditionally black areas to the south and west are being forced out as speculation drives prices beyond their reach. The visionary urban designer who came up with the original idea has already resigned in protest. All eyes are on what happens right now.

Eastside story: ‘It might change how people want to live’

The BeltLine is not a new idea. Ryan Gravel came up with the concept of turning the freight rail corridor into a transit loop while a masters student at Georgia Tech in the late 1990s.

Gravel spent his early life in Chamblee, just inside the I-285. The old town had been swallowed by suburban growth in the 60s and 70s and turned into a world of cars and malls. Growing up, he rarely ventured into the central city, but a student exchange trip to Paris opened his eyes to the heady freedoms made possible by public transit connections and walkable neighborhoods. The vision of the BeltLine was born.

Over many years, what started as a masters thesis was taken up by enthusiastic advocates and local politicians. The idea evolved to include a walking and cycling trail, and formal bodies such as Atlanta BeltLine Incorporated were established to press for its development.

Freight trains used the Southside (left) until a few years ago. The Eastside had been abandoned for decades

The first major section of walking and cycling trail opened in 2012. Paid for with private funds, the 14ft-wide ribbon of car-free concrete winds for 2.5 miles through Atlanta’s Eastside. The neglected, historically white neighbourhoods by the trail had already started to come up – evidenced by the conversion of the old Ford Factory into loft apartments – but the BeltLine really kickstarted development.

The derelict Sears warehouse was transformed into Ponce City Market, home to gourmet restaurants and boutique shops, office space and apartments. The eateries and stalls of Krog Street Market opened in 2014.

Sitting by the side of the trail with Gravel last week, it is clear he is proud of what has been achieved. We are passed by a constant stream of people: joggers and cyclists, dog walkers and parents pushing buggies. Some children walk and cycle to two schools just off the trail. Millennials zip by on the electric share scooters that now dot Atlanta.

It is a popular new social space in a city with no waterfront. A lucky few live along the line but the Eastside Trail has become a destination – a place people drive to, unload dogs or bikes, ride or stroll a couple of miles up and down, meet friends for dinner or coffee, and then drive home.

Common Ground on the Eastside is home to a brewery, restaurants and offices

Gravel hopes that over time fewer people will need to drive in, as the area around the BeltLine densifies and multistory parking lots are demolished to make way for more apartments, shops, restaurants and offices – but he is glad they are here.

“The BeltLine is bringing them in and they wouldn’t have been in the city otherwise,” he says. “They are spending money inside Atlanta and they are going home with a new appreciation of what’s happening here. It’s changing how they think about the city. It might change how they want to live.”

An outlay of $500m of mostly public BeltLine money has generated $4bn in private investment, with a further $4bn planned or proposed. New developments are springing up all along the Eastside, with developers tempting potential buyers with glossy billboards promising “BeltLine living today”, electric car charging points and the ability to “walk to shops and restaurants”. One under-construction development has townhouses starting at $1.2m.

A new south: whether you want it or not

I meet Angel Poventud, an enthusiastic cheerleader and long-time activist for the BeltLine, who offers to tour me around the complete loop, including the abandoned and active rail sections. It’s not strictly legal.

We start on the Eastside, where he is greeted by about half the people we pass. When we spot a group of schoolchildren on a bike tour, the teacher stops us and appears mildly starstruck – the class saw him in a promo video last week. Poventud gives them an impromptu talk and we’re off to the Southside.

Unlike the Eastside, which was abandoned decades ago, freight trains operated on the Southside section until a few years back. The city bought this four miles of land from freight rail company CSX last year and are hopeful of securing federal funding in December. If successful, they hope to start the Southside Trail by 2020. The MoreMARTA transport plan, agreed last month, may part-fund light rail in this section but more money is needed.

We nip past the “keep out” signs and cycle along the sandy trail where workers are busy tearing up the old track. Being a freight line, this section doesn’t connect with the communities around it – instead it either rides high above houses on an embankment, or is hidden down below in a cutting.

There are no access points yet, but that will change – and when it does so will the neighborhoods along this south side, some of the most neglected in Atlanta.

As we reported earlier this week, bright-yellow advertisements announcing “We buy houses” already hang on nearly every utility pole and tree, posted by home flippers looking to make a quick profit as prices swell.

Go west: ‘Is this for white people or black people?’

Soon we are on the Westside Trail – three miles from Pittsburgh to Bankhead, another troubled area.

In Mark Pendergrast’s book City on the Verge, Poventud recounts the story of a young black boy he encountered during construction of the Westside Trail who asked him who it was for – white people or black people. “I told him it was for everyone,” he adds.

The Westside Trail opened as a walking and cycling path last year. Federal money paid for this section and, unlike the cheaper construction of the Eastside, it is “transit ready” and features numerous access ramps, security cameras and lighting.

The trail becomes impassable with kudzu weeds after we reach the edge of Azalea Gardens – a neglected estate that will no doubt be demolished to make way for the trail and transit. The windows of two blocks are boarded up. A few people sit on their front steps, watching the world go by. We hear a staccato series of bangs in the distance, starting a debate on the street – “Fireworks or gunshots?” – until a man comes past confirming the latter.

Condemned housing in Pittsburgh, near the Southside and Westside sections

Rent here is cheap, but it is not officially designated as affordable. Where will these people go when the site is inevitably redeveloped?

The BeltLine’s original tax funding arrangement meant 5,600 units built along the route had to be affordable – but the reality is nowhere near that. Gravel resigned from the board of the Atlanta BeltLine Partnership in protest in 2016. “They had a target of 5,600 and they built 785,” he says. “About a third of those have now expired and are market rate. It’s terrible.”

He wants to see developers provide more affordable housing and be pushed to come up with better designs and more environmentally sustainable homes.

“We’ve been giving everything away for too long,” he argues. “We need to demand more. Now we have market forces behind us we don’t need to incentivize growth – we need to hold that growth to higher standards.”

Affordable units are part of the discussion but that’s not the be all and end all Atlanta BeltLine Inc CEO Clyde Higgs

On the 23rd floor of the Equitable skyscraper Downtown, the interim chief executive of Atlanta BeltLine Incorporated, Clyde Higgs, says he is “pretty confident” they will reach the 5,600 target by 2030. He says 1,600 affordable units have been built already, with a further 1,000 in the mile-wide general planning area around the loop. A new inclusionary planning zone means developments for rent must either offer 15% of homes at 80% average mean income, or 10% at 60%.

Plans were announced last month for 120 affordable homes in the renovation of the 1940s Capitol View complex in Adair Park. Higgs says he would love to ensure Azalea Gardens contains affordable housing – and that the ABI might buy the land and partner with a not-for-profit developer – but nothing has been decided. If a developer builds properties for sale, then they don’t have to include any affordable housing at all.

“Affordable units are part of the discussion, but that’s not the be all and end all,” he adds. “We want legacy residents to get opportunities. I think it’s going to be a positive thing but we need all the elements to be effective – we need the transit, the trail and the economic development. If we grow the pie, then residents of these areas have the opportunity for stronger incomes.”

Up north: the final piece of the jigsaw

A few days later I meet Poventud again to complete the loop along the active freight rail line, where he hopes future light rail will go. The walking and cycling trail will take a different route, on the other side of Peachtree Creek and along quiet streets.

In his day job as a train driver and engineer he rides this line every week, hauling lumber and steel, cement and chemicals out east to Athens and Gainesville. Some trains have 100 cars, a mile long or more, but he believes the line is underused.

“It’s a 100ft-wide right of way and we’re using 12 feet. We could be doing a lot more with what we have here. Because the trail takes a different route we could keep the freight active, and run light rail transit alongside.

“We have Piedmont Hospital on this stretch and it is one of the biggest employers on the BeltLine. Thousands of people work there and they have to rely on cars or infrequent buses. The rail line would give the workers and patients options.

“It’s the last piece of the jigsaw – and possibly the hardest piece – but it’s what everybody is focusing on now. We’ve got to close the loop. When it is complete you can come in at any point, and go out in any direction. It’s key to the system.”

Future Atlanta: ‘The answer cannot be to not improve things’

“The issue is how we manage this change,” says Gravel. “There are lots of tools available – affordable housing, zoning, other strategies – we know what we should do, but we’re not doing it. There is a lot of support for affordable housing in this city but politics means it is not happening.

“But the answer cannot be to not improve things. The answer cannot be to not provide transit and trails and green spaces, or develop schools and regenerate neighbourhoods.”

Another 2.5 million people are coming to Atlanta. Do you want them to live in a suburb, where the only way to get around is to drive? Ryan Gravel

After almost 20 years working to deliver his BeltLine vision, Gravel admits to being frustrated by the pace of progress. But he sees it as a generational change.

“The BeltLine is about building a future city,” he adds. “Transit is for everybody. It’s the part that gets people to jobs. I think when the transit is running, then all the people who currently don’t get it, will get it.

“And another 2.5 million people are going to come to the 20-county metro region by 2040. That’ll be 8 million people. Where do you want them to live? Do you want them to live in a suburb, where the only way to get around is to drive? The need for transit will be bigger than ever. It’s about what sort of city we want.”

Guardian Cities is live in Atlanta for a special series of in-depth reporting. Share your experiences of the city in the comments below, on Twitter, Facebook and Instagram using #GuardianATL, or via email to cities@theguardian.com