I am sitting at a table near the top of the highest skyscraper in the City of London with a plate of roasted octopus and a glass of Australian Riesling. It is 12.30am. Friday has just turned into Saturday. My vantage point overlooks the traffic jam I was stuck in earlier. Rows of red and white car lights tail back to the bars and clubs of Shoreditch to the north. It is a scene of busyness to rival the working day that ended some six hours ago.

My location is Duck & Waffle, a sleek restaurant on the 40th floor of Heron Tower that serves food 24 hours a day. Around me are youthful diners making a hubbub over background dance music played by a DJ. The Riesling tastes a little presumptuous — at £13 a glass, it is certainly that — but outside the lights of London twinkle into the distance. The night seems to have no horizon.

Or does it? In the 10 years since bar and restaurant opening hours were liberalised, London has taken strides to become the 24-hour city any self-respecting global hub should be. But the vision of 24/7 sophistication experienced in Duck & Waffle does not tell the full story. London’s attempt to be a city that never sleeps has run into obstacles.

The troubled gestation of the night Tube sums up the difficulties. The underground train service should have started running through the night in September. The scheme was announced two years ago by London mayor Boris Johnson at Piccadilly Circus station in the West End, the traditional centre of the city’s nightlife. With a flourish, he unveiled a map of the lines that, like New York’s subway, would have a 24-hour service.

But the night Tube has been put on hold, amid a dispute with staff over pay and working conditions. It is costing Transport for London (TfL), the city’s transport authority, £1.5m a month in pay for extra workers hired to run the service, although there is no hint of when it will start. Indeed, there have been whispers the project might be abandoned. Meanwhile, the underground continues to close at about 1am at weekends.

Getting around at night isn’t the only challenge. Up to 40 per cent of London’s small music venues have closed since 2005. Pubs and nightclubs are also shutting down, from venerable cabaret landmarks such as Madame Jojo’s in Soho to local establishments such as the Carlton Tavern in Maida Vale, illegally demolished by developers earlier this year. The city’s ravenous property boom is one cause of the closures. Venue owners also blame strict policing and local councils tightening up licensing laws. Are the lights being switched off on night-time London?

To test the city’s wobbling 24-hour credentials, I am pulling the proverbial all-nighter. My nocturnal odyssey begins in Dalston, east London, from where I walk to Shoreditch. It is 11pm, a time engraved on every drinking Briton’s mental clock as pub closing-time — or “chucking-out time”, as generations of frustrated topers called it.

Those over a certain age will remember the narrowing of drinking options after closing time. The holy grail was the illegal pub “lock-in”, a furtive after-hours drinking session conducted behind locked doors by a disobedient landlord. Folk wisdom held that you should pour your pint on the ground if the police entered: the beery puddle would supposedly not incriminate you.

The Licensing Act 2003, introduced in November 2005, dealt the lock-in a blow, liberalising opening hours that dated back to the first world war. The ever sober British press warned of an epidemic of drunkenness and disorder. “Terrible toll of this binge that never ends,” a Daily Mail headline prophesied in 2004. But levels of drinking and violence, which were already declining, have continued to do so.

A couple in the street outside Cargo, a Shoreditch nightclub

Almost every pub I pass on the way to Shoreditch is open after 11pm. Traditionally a working-class neighbourhood, the area has over the past decade been gentrified into a thriving night zone. On Rivington Street, I find DJ Darby, 41, standing outside a venue where he will later be playing at an “old skool meets new skool” night. A local, he welcomes the changes.

“Twenty years ago, 10 years ago, round here was a shithole — excuse my language,” he says. His friend, Marcus, 36, also a local, weighs in: “It’s got trendier and more multicultural: people from Spain, Italy, India, everyone. Everyone’s happy-go-lucky, there’s no trouble.”

“Ten years ago, if you headed down to London Fields, you’d get rushed,” DJ Darby adds, referring to muggings in a nearby park. “Now you get, like, a latte.”

Down the road at Cargo, a bar and club under Victorian railway arches, a queue has formed. Those entering have to produce photo ID, which is scanned as though at an airport. Staff in black paramilitary garb man the entrance; several more observe from over the road.

A young man in the queue, who gives his name as “Dr X”, denounces overzealous security as the worst aspect of London nightlife. “It’d be better if the arseholes weren’t such bouncers,” he says to looks of bemusement from his companions. “Um, I mean if the bouncers weren’t such arseholes.”

…

The night economy is worth £66bn annually to the UK, according to pressure group the Night Time Industries Association (NTIA). Its chair, Alan Miller, estimates London’s contribution to be at least one-third of the total. “Twenty-four-hour cities are the cities of the future,” he says. Yet he believes the sector is being held back, blaming “a combination of over-regulation and a precautionary outlook that holds venues, clubs and bars responsible to a degree no other industry faces”.

Take lost mobile phones, for example. If someone reports their mobile as stolen in a club in order to claim insurance, the crime figures tick up a notch and the venue is held accountable. A disorderly reputation can lead to police recommending restricted opening hours or even closure.

In the Bussey Building, Peckham

Earlier this year, the Metropolitan Police commissioner Sir Bernard Hogan-Howe argued that the number of pubs and clubs should be reduced to lower alcohol-related violence. Miller bridles at the idea. “If someone is mugged in a shopping centre, no one is going to propose shutting it down,” he says.

Matthew Beaumont, author of Nightwalking: A Nocturnal History of London, says the fear of youthful nocturnal revelry “goes all the way back to the Middle Ages and the figure of the apprentice. In the daytime they were effectively indentured to a master and supposed to obey a strict curfew. But if they could get out, then they tended to be pretty rowdy. They’d go to the centre of the city to carouse in pubs and do things like climb the towers of churches and ring the bells in the middle of the night. They were a constant thorn in the side of the authorities.”

During my own night walk, almost all the people I encounter are in their late teens or twenties. Daytime London, with its soaring accommodation costs, is increasingly inhospitable to them. Night is their zone. Yet it, too, is being squeezed. Hackney Council, the local authority for much of Dalston and Shoreditch, this year proposed a new licensing policy shutting bars and clubs by midnight at weekends. Noise complaints in the borough have been rising and the council has a legal duty to investigate each one; it has no such duty towards the interests of bar-goers and clubbers.

The NTIA would like London to have a “night mayor”, an official to mediate between the competing interests of the city at night. This happens in Berlin and also in Amsterdam, where Mirik Milan performs the role, having been elected by a panel of nightlife representatives — “people who run live music venues, dance clubs, gay Amsterdam, maybe the squatting community”, says Milan, sounding the quintessence of Dutch tolerance (his email sign-off reads “Rebel in a suit”). He assures me he does not want to upset Amsterdammers hoping for a solidly bourgeois good night’s sleep. “We try to build a bridge of trust between venues and residents,” he says. “It’s not like we’re saying: ‘We need more nightclubs, we need more parties.’ There’s an up- and a downside, and we have to look at both really clearly. Night culture is like an ecosystem. How can you maintain a culture if you have no idea what is going on?”

…

Back in Shoreditch, I hail a taxi to Duck & Waffle, only to find myself stuck in that traffic jam. “The traffic is always busy now,” says Jama, the driver. “It’s the same even at 3am.”

The idling cars are a sign of a thriving nightlife but also of a dysfunctional transport system: TfL estimates 180,000 journeys will be made on the night Tube between 12.30am and 6am, if or when it begins. Half are expected to be newly generated journeys, the rest will be by people who previously used night buses or taxis.

By the time I emerge from the restaurant, it is after 1am. Having missed the last Tube, I travel south of the Thames with Uber, the ride-hailing app. The service is facing restrictions from TfL, including a stipulation that drivers must wait five minutes between jobs before picking up new passengers. “It’s already so busy, it’s going to get crazier,” says Mudassir, my Uber driver.

Peckham is a scruffy south London district dubbed the “new Dalston” for its developing nightlife. My destination is the Bussey Building, an arts centre and clubbing space in a late-Victorian warehouse. A planning notice by the entrance spells out the danger: a proposed new development of flats and shops on the site. “It’s madness,” says Mickey Smith, who opened the Bussey Building eight years ago. “I find it perverse, the idea of risking Peckham’s cultural future by building luxury flats that no one locally can afford to live in.” It is part of a wider trend. Since 2005, the number of nightclubs in the UK has fallen from 3,144 to 1,733.

In the West End, a busker strums a guitar, a drunk staggers along; one lad dives on to a pyramid of bin bags

Inside there are two dance floors, one with a sparse audience observing a DJ play moody electronica, the other crammed with people dancing to techno. At one point the DJ spins an old-fashioned crowd-pleaser: 1979 disco stormer “I Can’t Change It” by Linda Evans. The music will go on until 6am.

The atmosphere is friendly: I see no bouncers who are arseholes, nor vice versa. Maureen, a staff member in a high-vis tabard, is pessimistic about the threat to the Bussey Building. “Even despite people protesting I’ve noticed a lot of things in south-east London being shut,” she says. Daisy, 23, claims “south London doesn’t need any more gentrification”.

If the Bussey Building closed, “We’d be out on the streets drinking like tramps!” says Kristina. An even worse fate occurs to the moustachioed Luke, 23: “We’d have to go north of the river . . .”

…

An Uber ride later I am in Vauxhall, a central south London riverside district, which has become a clubbing destination, especially for gay people. Outside the Royal Vauxhall Tavern, south London’s oldest gay venue, I meet William, 24, who is dancing to the sounds spilling out of the Eurovision-themed event inside. The venue, under threat of closure, won Grade II-listed status this year after a campaign to save it. “Being listed isn’t really being saved at all,” he warns. “This building could have anything in it, it doesn’t have to have a gay bar in it.”

Bar Italia in Soho

Nearby, in one of the blocks of flats that have sprouted up by the Thames, a 24-hour gym is empty except for a cleaner lugubriously pulling an industrial vacuum cleaner back and forth. New developments can have a parasitic relationship with London’s nightlife, capitalising on an area’s “buzz” to drive up prices, but then sterilising it with the wrecker’s ball or noise complaints from newly installed residents.

By 4.45am, I have reached the West End. A busker strums a guitar in Leicester Square, a solitary 50p coin in his case. The odd drunk staggers along; one lad dives on to a pyramid of bin bags in Soho. Near the burlesque stalwart Madame Jojo’s, now boarded-up, an insalubrious young man searches for the remnants of the area’s sex industry. “Hey,” he says, lunging towards me, “Where’s the bitches at?”

My final destination, reached by an empty night bus that speeds along empty roads at 5.30am, is a restaurant that claims to be London’s oldest 24-hour eatery. Vingt-Quatre is on Fulham Road in moneyed west London. A pastiche zinc bar with doormen, it is a throwback to the old London of early closing and late-night drinking dens.

A near-deserted Piccadilly Circus

A large man who looks like an arms dealer eats hamburger and chips with a younger woman who picks at a club sandwich. Two drunk men order one of the cheapest items on the menu — fruit salad — to go with their cognacs. A man in a bowler hat seems to have stepped out of A Clockwork Orange. Another waves a bottle of pink champagne munificently.

It is 6am. London’s night has contracted to the dimensions of the room, its last dregs eked out by a curious medley of desperadoes. In the spirit of 24-hour London, I drink a bottle of beer and eat a congealing poached egg from the breakfast menu before catching the morning Tube home.

Ludovic Hunter-Tilney is the FT’s pop critic

Slideshow photographs: Harry Mitchell