On Monday, Radio 1 cleared out its normal news schedule for a series of special reports – the sort they normally reserve for a rock-star death or a general election. The breaking news story they were covering? The death of the British nightclub. “These clubbers have nowhere left to go,” said the host in the grim tones of a Comic Relief appeal, before going to a reporter in Peterborough wandering around shut clubs as if Hurricane Killjoy had just torn Britain’s city centres asunder.

How can Britain possibly be going through a nightlife crisis? Our popular culture has never been more focused on dance music and getting smashed. Local news is overrun with bacchanalian images of wild town centres and police struggling to contain punch-drunk revellers. House music, once a counterculture heard only in raves and on pirate radio, has long been the sound of the British high street. Music festivals have become as ingrained in our culture as Christmas, an annual trip to a field to take drugs and watch Chemical Brothers now a rite of passage on a par with leaving a glass of milk out for Santa. Just two months ago, a report described the booming health of Britain’s night-time economy, claiming the sector was worth £66bn. Six per cent of the UK’s domestic product is generated by night-time businesses, which employ around 1.3 million people, up by 500,000 from 2002.

Which is why some people were shocked this week when the report released to Radio 1 – from the Association of Licensed Multiple Retailers (ALMR), the group that represents venues – revealed that more than half of UK nightclubs have closed in the past 10 years. ALMR’s chief executive, Kate Nicholls, said that, in some towns, “they are gone for good and we’re never going to get them back”. To those who have been trying to go clubbing in Britain, it will not come as a surprise.

It might seem that this is a story of interest to a few druggy hedonists, but the deterioration of Britain’s nightclubs has been ruthless. Clubs are the most vulnerable victims of gentrification because they are difficult to defend, seen by people who don’t go to them as a public nuisance. In the past few years, hundreds of nightclubs have been shut in increasingly confrontational circumstances.

Two years ago, 175 Metropolitan police officers raided the 250-capacity venue 93 Feet East in dramatic scenes that looked like something out of Iraq war film The Hurt Locker. They arrested only nine people, mostly for minor drug-possession offences, but the club had its licence immediately revoked and was closed for months, only reopening after a court overturned the decision. In Glasgow, police put onus on the Arches to comply with incredibly strict searches and security measures or face being shut down. The club said that they complied with everything asked of them, but the police still persuaded the council to limit the club to a midnight licence. This effectively made it financially impossible to run the club and the huge community arts centre that ran during the day. In north London, People’s Club, which served African-Caribbean communities as well as holding Eternal, an old-style rave that hosted secret sets from Jamie XX and Evian Christ, was closed earlier this year after a long-running battle with Islington council. It ended up paying the council £10,000 in costs after it appealed the decision to revoke its licence. There are so many stories likes these, where clubs are treated like a hooded teenager standing outside a shop, the authorities looking for any excuse to move them along.

“It’s mostly the smoking ban, in combination with a far lower tolerance of noise pollution,” explains Andy Peyton, the promoter of XOYO, one of the last big clubs in London, when I ask how clubs are caught out. “One single noise complaint can lead to your licence being in jeopardy. It will never be ignored, however weak that noise complaint might be. The XOYO smoking area is tiny, but there’s really no way to expand it. To have people smoking out front would end the club overnight.”

Ravers at Liverpool’s Cream nightclub. Photograph: Naki/PYMCA/REX Shutterstock

So, back to that paradox. With British nightclubs in peril, how can the night-time economy be growing? Look at any city centre and the answer is obvious. Where once there were clubs, now there are cocktail bars, fancy pubs with late licences, small-plate restaurants and gourmet street food. Now that most places shut at 1am, no one feels the need to stay out to 6am.

It would be a mistake to lay the blame for club closures solely with the powers that be. The sadder truth is that the appetite for this kind of fun has diminished. When you’re paying £9,000 a year to go to university, it becomes harder to justify going out every night. Many young people in cities have to settle down with a partner sooner than they might have done, in order to share a bed and split the rent. And when you’re living with your partner, you’re probably more likely to go to the cinema than a techno night. Youth unemployment is at a 20-year high, which means there’s less money to spend on going out, and those with work are often in low-paid roles that force them to take a second job, or have to work unsociable hours. For many, clocking off at 5pm on a Friday and being out of action all weekend isn’t an option. Much safer to go to the pub, sink a few craft beers and save your big blowout for a festival in the summer.

“This year, we’ve had the most festivals ever in the UK,” says Rob Casson, a manager at ticketing agency Skiddle and the promoter of Freeze events in Liverpool. “People who buy their tickets for those in February or March can’t afford to go out clubbing. If you’re a student, unions put on very good parties which are really cheap. A pub can have a DJ in the corner with decent music and it’s free and people are quite happy with that. The days of going to your usual club are gone.”

Casson books dance-music events in unusual venues – the Liverpool Cathedral, old psychiatric institutes – which he says are doing a roaring trade. A lot of his events are in the daytime, running until 10pm, with a “part two” in a club for those who are keen. Across the country, it’s a similar story, with one-off warehouse parties and day festivals booming while clubs suffer.

But nightclubs play a role in society that’s very different from warehouse parties or festivals. They provide a fixed location where people you know and people you would like to know come back every week. Clubs can provide an incredible sense of solidarity for anyone, but if you’re gay, from an ethnic minority or just a social anomaly, nightclubs can be a lifeline of unimaginable importance. Fun, in this sense, is not just blind hedonism, it’s security, escapism and a feeling of belonging; it’s hard to think of anything else that can provide that. And the things that emerge from it can be spectacular.

“It used to be that scenes would begin in a particular spot, with the Blue Note [now closed] and drum and bass, or Plastic People [now closed] and dubstep,” Joe Ray, from British dance duo Nero, tells me. “But if you look at deep house – the latest sound to cross over – that’s not really come from anywhere; it has no home.” Peyton says the same thing: “Without nightclubs, you lose the chance to develop scenes and you lose the chance to experiment. With our residencies, we can put on people who will not necessarily sell tickets on their own, but we think are great, and nurture them.”

Fabric, in central London, which has had to introduce airport-style security at its doors. Photograph: Alamy

I put this to Casson, who is rather bullish about this new era in nightlife, and even he agrees. “I think that’s probably true; we’ve tried riskier lineups with our nights and it has been a disaster. We lost £14,000 on one night with acts that had never played Liverpool before. That’s a lot of money for a small company. We have to be careful and make sure we’re booking bigger names.”

Councils aren’t just shutting down licences willy-nilly. What they are doing is limiting the capacity for fun. They’ll say clubs can only trade until midnight. At Fabric, in central London, there is now stringent security, and anyone caught with drugs is bundled into a police van. Before the Arches closed down, one measure suggested by police was “moments of calm”, where DJs stop playing music for five minutes each hour, with the lights switched on. The idea seems to be that if you regulate nightclubs to such an extent that the fun is squeezed out, they’ll close down anyway, or be so anaemic as to no longer be a bother.

The unspoken truth in all this is that the law hasn’t changed much – it’s just that whereas once the authorities seemed to turn a blind eye to venues with questionable licences, now even the smallest failing on a club’s part can get it closed down. “Honestly, we really panic if a mobile phone has been reported stolen at the end of the night. We have to find the phone ourselves, basically, or face a lot of pressure from the licensing authorities. No question, one phone theft can put your licence in jeopardy,” says Peyton.

The whole point of a nightclub is that it’s an experience greater than the sum of its parts – a big empty room with someone playing loud music should get boring pretty quickly – but it’s made by the people, the fashion, the drugs and the frisson of possibility that comes from a night without rules. It’s why Glastonbury is more fun than V Festival, why camping is more fun than a night in a Travelodge and why people traipse to clubs in the middle of nowhere to see DJs. A night out is not like a massage; you can’t just pay your money and feel better after 45 minutes; there has to be some kind of ineffable experience. The death of the nightclub is certainly not the end for nightlife, but it does permanently change the sort of fun you’re allowed to have. It’s hard to believe that the outsider communities that dance music has always engendered would ever find a home at a city festival sponsored by a pear cider.

Raving at Garlands, Liverpool. Photograph: PYMCA/REX Shutterstock

‘The authorities are the biggest challenge to clubs’

“The authorities are the biggest challenge to clubs. The police are concerned about their lessening resources due to cuts, and it’s an easy way to say that crime has been decreased by putting pressure on the night-time economy. Sir Bernard Hogan-Howe recently said that 50% of bars and clubs should be closed to reduce crime. There’s a notion that we should be responsible for any individual act that takes place outside the premises. Unlike a bank robbery – where the bank is just seen as a victim of crime – nightclubs are held accountable for things that happen both at their premises and outside.”

Alan D Miller, founder of the Old Truman Brewery in east London and chairman of the Night Time Industries Association

“Yes, the internet plays a vital role in how people communicate now, it’s less organic and physical. Before, you’d make friends at clubs, whereas now people meet through social networks. But the desire to lose your mind to music is still very much there, the options are just limited. The internet hasn’t killed subculture. Online platforms like Boiler Room, that push the best new music, are a new kind of club night.”

Jennifer Byrne, producer of Dazed’s Music Nation

“There’s been a tendency for the major, risk-averse nightclubs to adhere to the lowest common denominator. Commercial nightclub operatives have been uncreative and too concerned with catering for stag nights, hen dos and office parties. Once upon a time, people would have gone to clubs to hear exciting interesting music, to experiment with fashion and seek out new attitudes and ideas; now people get an entirely predictable experience. I know people who go all the way to Berlin or Amsterdam because they know the local disco round the corner is boring.”

Dave Haslam, author of Life After Dark: A History of British Nightclubs and Music Venues

“Most areas in town cities are filled with middle-class residents, they know the sound legislation is draconian. You make a sound complaint, I will have a council officer here within 20 minutes. So many clubs in Soho were closed because of blocks of flats being built; Egg in Kings Cross, I doubt they’ll survive for more than three years because of the block of flats being built opposite. What are our city centres for? Why did people come to London? It wasn’t because of the traffic and taxis. It was because we have a brilliant world-famous nightlife. Town centres are for fun; they should be noisy, vibrant places. You can’t have 50 residents who are in houses that were £300,000 , are now £2m, wanting their houses to cost half-a-million pounds more, wanting to close clubs down.”

Alex Proud, founder of the Proud Group

“Clubbing is a movement that brings people who share the same musical connection together. I loved clubbing so much I started my own club night to unite people with the same music taste and to make it easier for people to see all their different friend groups in one room. Under this current government, we are made to work so much that seeing your mates becomes hard work. You need a designated club that you go to where you can let your hair down in the comfort of your friends.”

Panashe Nyandoro, a club promoter in north London (House of Camden)

Additional interviews by Nadia Khomami

• This article was amended on 14 August 2015. An earlier version inaccurately described the security at Fabric as “airport-style”.