On the evening of 18 November 1987, a lit match fell between the cracks of an old wooden escalator in Kings Cross underground station, landing on decades of accumulated grease, grime and dust. The resulting fire killed 31 passengers below ground. Until Grenfell Tower it was the most serious fire in London since the Blitz, but the catastrophe was also a wake-up call for the city’s mass transit system. After an investigation, the top two senior London Underground and London Regional Transport officials resigned.

Across the pond in New York City, meanwhile, the subway system was in rude shape. Government investment was flooding in. Stations were being renovated; subways cleaned of graffiti and decay; and years of deferred work finally being addressed. Ridership responded in kind, rising by 7.4% between 1984 and 1994.

Today, the fates of the two networks have reversed. The London Underground has become one of the world’s most reliable and innovative metro systems, while in New York a state of emergency was declared in 2017 as the subway system achieved the dubious distinction of the worst on-time performance rates of any major metropolis in the world. Videos of ceilings collapsing and stations flooding went viral.

The burnt-out remains of the ticket hall following the Kings Cross fire in 1987. Photograph: Nils Jorgensen/Rex Features

The story of how the two most important metros in the western world, serving cities with such similar pressures of population (between 8 and 9 million, and growing) and hyper-development, diverged so drastically is a story of many things – but chiefly, says Nicole Badstuber, an expert who studies transport governance at University College London and once worked for Transport for London (TfL), it is “a story of maintenance”.

“Basically since the end of the second world war, there was no investment” in the London transit system, she says. The fire was an explicit message that new cash was needed immediately. So, too, was image: the Thatcher era saw London’s new role as global banking capital find its home in the London Docklands, where the towers of Canary Wharf were rising.

The new pressure resulted in two new high-tech arteries to serve Canary Wharf: the Docklands Light Railway (DLR) and the Jubilee Line tube extension. Suddenly British developers were convinced of transport’s role in expanding wealth generation.

A 2017 transport protest in New York in 2017. The same year, ‘a state of emergency’ was declared over the city’s ageing subway. Photograph: Kathy Willens/AP

“There was no way [developers] could get this amount of workers without having mass transit,” says Badstuber. “That is important because you not only had a business lobby and a developer lobby starting to say, ‘We need to invest in transport,’ but also, you now have a separate level of clientele that’s using the network, and has certain expectations.”

You need a large amount of capital investment in the system. That’s what TfL got, and that’s what TfL needed Nicole Badstuber, University College London

That same lobby also endorsed the idea that transit should be governed by the city, not the national government, an idea that was overwhelmingly passed by referendum in 1998. Transport for London (TfL) was born, and the newly elected mayor, Ken Livingstone, whose fierce advocacy for public transport and equity was crucial in getting the agency created, put a majority of his office’s budget into transport.

With main arterial roads under its jurisdiction, TfL successfully revamped the city’s long-ignored bus system: it created bus-only lanes, introduced flat fares and increased service. It also poached the head of the New York subway, Bob Kiley, and by the late 2000s it had taken control of all London Underground lines – previously controlled by companies through public-private partnerships.

Nearly two-thirds of all trips in London are now made on foot, bicycle or public transit. Photograph: Dylan Martinez/Reuters

“These companies were not bringing the investment that was expected, particularly to infrastructure,” Badstuber says. “To me, this is all leading up to a realisation that, actually, you need a large amount of capital investment in the system. That’s what TfL got, and that’s what TfL needed – and to me that’s what any large system needs.”

Crucially, central government also committed to new funding. There were two consecutive five-year cash injections to upgrade signalling, rolling stock and stations, and an expansion of the Overground, the surface-level train line that now forms a loop around the entire city, integrating much of east and south London. Livingstone’s signature 2003 transport initiative, congestion pricing, also helped pad the purse. Londoners struggled through years of night and weekend closures, but the modernisation work began to pay off: ridership hit record highs, the trains ran on time and the city became known worldwide for its gleaming tube. In 2008, TfL won an award for best metro in Europe, and the 2012 Olympics saw the culmination of an entire new set of plans for transit-oriented development at Stratford.

Nearly two-thirds of all trips in London are now made on foot, bicycle or public transit. The goal is to increase that “modal share” to 80% by 2040, according to TfL’s director of city planning, Alex Williams. “Since TfL was formed in 2000, when just over half of all journeys were by sustainable modes, billions of pounds have been invested into London’s public transport,” he says. “This has included creating a greener, efficient bus fleet, upgrading the Tube with faster, more frequent and reliable trains and making stations more accessible for our customers.”

A 540-tonne boring machine breaks through the earth south of the Thames after completing a tunnel which now links the Docklands Light Railway to Woolwich as part of an ambitious expansion project. Photograph: David Levene/The Guardian

In New York, the story took a very different turn. Although it has similar modal share figures to London, the subway system has now spawned entire sub-genre of reporting spawned to ask the question: “What happened?”

An excellent investigation by the New York Times in 2017 pointed to two key factors. First, in the 1990s, elected officials began a pattern of diverting maintenance funds to other political priorities. Second, too much has since been spent on vanity projects and consulting fees, leaving the system starved for cash. In 2019, only two of New York’s 27 lines have modern signal systems; in London, half of the system is online, with the rest expected by 2023.

Mass transit is not a priority for the federal government Mitchell Moss, NYU Rudin Center for Transportation

But it’s also an issue of governance. The two systems governing the respecting metros are very different. Unlike TfL, the Metropolitan Transportation Authority (MTA) is a holding entity for separate operating companies (including the Long Island Railroad and Metro-North); unlike TfL, it doesn’t control street operations; and unlike TfL, it doesn’t answer to the city but to the governor of New York state, which many critics say leaves it too vulnerable to politics. The longstanding blame-game between the mayor, Bill de Blasio, the governor, Andrew Cuomo, over who’s responsible is a testament to this dysfunction.

Moreover, the federal government’s financial contribution, says Mitchell Moss of the NYU Rudin Center for Transportation, is “modest”, and usually goes to huge capital projects such as East Side Access, which will link commuter rail to Grand Central Terminal. “Mass transit is not a priority for the federal government,” Moss says. “The federal government is very involved in airport construction and highway finance, but not mass transit. And that’s a key point.”

In other words, the MTA isn’t getting funds from Washington to upgrade signals anytime soon. The city doesn’t have the same national role as London, which generates nearly a quarter of the UK’s GDP, nor does it see more than a million passengers arrive from outside every day. In the US context, New York’s transit problem is New York’s to fight alone.

A train passes through the Bronx in 1973. Riders turned their backs on the city’s run-down subway in the 1970s, but investment in the 1980s saved the system. Photograph: Jon Naar

Moss argues, however, that hitting rock bottom is a natural point in a 30-year cycle. “We get a big infusion of investment … which we had in the 1970s and 1980s, and now it’s run its course,” Moss said. “Now we have to put more investment in.”

That’s precisely what is happening, in the form of an immediate jolt of operational money (the Subway Action Plan) and a nearly $40bn (£30bn) long-term modernisation scheme (the Fast Forward Plan). Together they represent an unprecedented new addition to the MTA’s coffers, and there are encouraging signs of success: the subway’s performance recently achieved a six-year-high with more than 80% of trains arriving on time, a payment system based on Oyster will be rolled out system wide by 2023, and a new plan for a London-inspired congestion pricing scheme will give the agency a dedicated stream of funding.

“Transit has been in crisis before,” writes Andy Byford, the MTA’s British president (who cut his teeth at TfL), in the plan’s foreword. “In the 1970s, system ridership was in steep decline. New Yorkers turned their backs in droves on a dirty, graffiti-scarred, dangerous subway where service breakdowns had become the norm.” The 1980s action plan saved the system once, Byford argues, and the new action plan can do it again.

The Second Avenue subway, which opened in 2017. Photograph: Alba Vigaray/EPA

But the system has a lot of catch-up to play. Take expansion, for example: in four decades it has added just four new subway stations (the 7 train extension and the first three stations of the Second Avenue subway, which saw the most expensive track-laying on Earth). As the population and attention of New York City increasingly leaves Manhattan behind and moves outward, does the subway need to keep growing, too, to save itself?

Moss argues that there are alternatives to expansion, such as cycle hire, rideshare apps and ferries. “The key challenge the MTA faces is to maintain and improve a 100-year-old system, rather than expand it at the price of maintenance,” he says.

Indeed, London’s ambitious expansion could spell trouble in the years ahead. The delay and exorbitant cost (£18bn and counting) of Crossrail has diverted money from other important projects, such as installing new trains, expanding cycling infrastructure and improving bus lanes. Recent delays due to faulty trains on the Jubilee line feeding Canary Wharf could be a harbinger of things to come.

Immediate issues of capacity, staffing cuts, and dangers to public health are also making headlines. A decline in bus ridership since 2014 has led to service cuts, which largely fall on low-income riders and students who can’t afford the Tube, undercutting mayor Sadiq Khan’s equity initiatives like the Hopper fare (a single fare regardless of how many buses you take in a trip) or fare freezes, Badstuber argues. Nor, with an uncertain political future, can TfL be as reliant on Westminster for help. “I think London is potentially at the cusp of losing its way a bit,” says Badstuber.

In other words, the lines may be converging again, and the metro systems of London and New York may soon find themselves at a crossroads much like the 1980s: a time when the decisions taken now will be felt for a generation.

This article was amended on 16 December 2019 to correct a sentence which referred to the DLR as a monorail. It is, of course, a largely automated light metro system.

Follow Guardian Cities on Twitter, Facebook and Instagram to join the discussion, catch up on our best stories or sign up for our weekly newsletter