“It started two years ago. We’d been in business two decades, working with planners, licensing authorities, and the police. But then all these weird things started happening.”

Alan Miller, a 44-year-old serial entrepreneur and devoted lover of London, is sitting in the Blues Kitchen, a luxurious deep south-themed bar and restaurant in Shoreditch. Around us, people are starting to congregate in readiness for a night out, and the early-evening air is abuzz with the sense of quiet excitement that tends to arrive in British cities in the summer. In the surrounding streets, gaggles of people are drinking outside, and viewed from a certain angle, the scene might look idyllic: proof, perhaps, that the UK is at last embracing the continental ways that have long been said to be beyond our national grasp.

At a huge table nudging an open window, Miller starts to explain the turn of events that have defined the last three years of his life. He talks quickly, with wide-eyed passion, and his spiel bounces wildly between high and low culture – from Shakespeare and Plato to Liam Gallagher – and past a huge array of reference points. But he soon settles into what he has come here to talk about: what he sees as the strangling of too many UK bars, clubs and music venues by oppressive policing, and the essentially Victorian attitudes of local councils.

Miller is part of the generation that was turned on to entrepreneurialism and the creative possibilities of bars, clubs and music by the great acid house upsurge of the late 1980s. In early 1995, he was one of the people who redeveloped the old Truman Brewery on Brick Lane, turning an empty industrial site into a honeypot for small creative businesses, and attracting crowds of people from across the capital. “Some people thought we’d lost our marbles,” he tells me. But the idea worked – and among the proofs of this was the success of the Vibe Bar, a 450-capacity venue that put on regular events, showcased new talent, and became a byword for the transformation of east London.

As of 2012, though, the demands he faced from the Metropolitan police and Tower Hamlets council’s licensing department began to get more and more onerous. He had to commit to employing specific numbers of security guards, irrespective of his own judgment. His licence was made conditional on the installation of an ID scanner, which stored the identities of his entire weekend clientele, supposedly because it reduced the chances of any trouble. The final blow, he says, was the imposition of restrictions including a curfew, which meant he lost 30-40% of his trade in just two years.

In 2012, the Met launched Operation Condor – what the force’s own publicity called “a massive crackdown” on almost 5,000 licensed premises across London. By the spring of 2014, the Met’s commissioner, Sir Bernard Hogan-Howe, was suggesting that a new drive against supposedly alcohol-related crime would have to take precedence over any idea of regenerating areas – as happened in Brick Lane – via bars and clubs. “We need to make sure there is good control of the supply of alcohol,” he said. “This means licence numbers, density and licensee-regulation being a priority for local authorities, however much they would like to develop their local economies.”

‘You end up in a constant panic’, says Alan Miller, bar owner and chair of the Night Time Industries Association, of today’s ever-changing licensing requirements. Photograph: AIecsandra Raluca Dragoi for the Guardian

All this, inevitably, sent Miller and a lot of his fellow proprietors into a spin. “You end up in a constant panic,” he says. “A good example is what happened with the ID scanner. We were in situations where we had to have it on from 7pm on a Friday: that was mandatory. And it would be playing up, and the police would be there going, ‘Well, we might have to go to a licensing review.’” He also claims he was upbraided for not bringing the police sufficient numbers of his clientele who had been caught with drugs. (When I put this last allegation – and others – to the Met, they failed to directly respond, instead giving me a statement insisting that licensed venues are “policed appropriately according to intelligence and demand” and that the force has “a responsibility to keep people safe and we want people to enjoy their evening out in a secure environment”.)

The Vibe Bar closed in November last year. In the meantime, Miller has been amassing similar stories. Earlier this month, for example, the Arches in Glasgow – a renowned venue that included a theatre, restaurant and café – closed its doors, after police complaints about crime and drug use convinced Glasgow council to force it to close at midnight, which killed its club nights and made it financially untenable. The story highlighted an increasingly familiar plotline, whereby places praised by politicians and sometimes assisted by public money fall foul of other branches of officialdom: as Glaswegian musician Stuart Braithwaite, of the rock band Mogwai, put it, the Arches was “subsidised by the government and yet strangled by the authorities”.

This is the kind of tension that Miller wants to blow open, using a new organisation called the Night Time Industries Association (NTIA), whose membership now includes around 170 businesses and organisations. This week, it launched a report titled Forward Into the Night, authored by the contrarian academic and sociologist Frank Furedi – who, in keeping with his habit of decrying the modern fear of imaginary dangers, writes about the hours after twilight as “a focus for moral anxieties and social insecurities”. Rather more usefully, he also points out the worth to the UK economy of night-focused businesses, whose annual turnover comes in at around £65bn, and which account for nearly 6% of national income.

The people who run bars and clubs are often reluctant to talk, for fear of irritating the authorities on whom they depend for their licence. Off the record, one big player in London’s night-time economy tells me about the endless perverse consequences of what has recently happened. Installing ID scanners and metal detectors, he says, is hardly the best way of creating the relaxed atmosphere most conducive to running a successful and peaceful bar. Panic about Romanian crime gangs, he goes on, has meant that some establishments now see a passport from that country as a reason not to let its holder inside, despite the fact that most Romanians will obviously not have any criminal intentions.

In some London boroughs, he tells me, police now operate a “last drink” policy, which means that if someone commits a crime deemed to be alcohol-related (something that need only be “perceived” by the victim), officers will find out the last place they had a drink, and often pay it a visit, sometimes with a view to questioning its licensing arrangements.

Most farcical of all, he says, is the tangle of issues around lost mobile phones. Their owners often report them stolen for insurance purposes, which necessitates naming a particular venue, and adding to local crime statistics, tweaking the attention of the police. In order to reduce the chances of this happening, he says, he now offers his glass collectors £50 rewards for finding mislaid phones, and reuniting them with their owners.

Miller is well aware of this last issue. “For no reason at all,” he says, “all of a sudden, a bar or club is the location of a crime. The borough police commander is under instruction to reduce crime numbers, and there seems to be a crime spike in these places. And then you end up having all these conversations: ‘Look, crime is increasing all around you.’”

Miller calls an Uber cab and we head from Shoreditch towards Dalston. On the way, we talk about elements of the night-time economy that might be complicit in their own woes. Crackdowns on intelligent, responsible venues such as the Vibe Bar, I agree, look misplaced. But what of those places whose owners – corporate pub companies, very often – rip out any furniture and then play music at conversation-stopping volume, so as maximise their punters’ alcohol intake? The result, very often, are the scenes that might not materialise in the more upscale parts of east London, but that millions of us are familiar with: by the wee hours, a great staggering cavalcade of people, some of whom think the ideal conclusion to their evening is a random ruck, which may or may not end in the glare of a flashing blue light.

A gig at the now-closed Arches in Glagow, which was ‘subsidised by the government and yet strangled by the authorities’. Photograph: Alamy

“There are different places for different things,” he answers. “Sometimes having a load of drinks with your friends can be fun … I think there’s a personal-responsibility conversation, and if people are stupid or break laws, they should be held accountable. Not the industry.” This, I suggest, might not sound all that convincing. “One of the prices of freedom is that people may do something crazy,” he replies. “The question is, who’s responsible for that? And does everyone have to lose their freedoms in order to prevent it?”

Miller and his allies have stronger arguments in their favour, mostly bound up with social trends that suggest the current panic about drinking might be increasingly misplaced. The NTIA report quotes findings from the Office for National Statistics that the proportion of young people who have indulged in “binge drinking” in the last seven days has fallen by more than a third since 2005, from 29% to 18%. A quarter of 18- to 25-year-olds do not drink at all. The number of alcohol-related violent incidents fell from 1.1m in 2004-5 to 700,000 in 2013-14. And even if London is under the metaphorical cosh, other cities’ experiences suggest that expanding the number of licensed premises does not entail an explosion of chaos: the night-time economy of Manchester – which campaigners hold up as an example of authorities and businesses working productively together – has grown significantly over several years, but levels of violent crime in the city centre have fallen by 8.2% over the past three years.

Ten minutes after leaving the Blues Kitchen, we arrive at Rotorino, an Italian restaurant and bar on Kingsland Road in Dalston. At one of its al fresco tables, I meet its co-founder, Jonathan Downey, 49, a relocated Mancunian – whose life, he says, was changed for ever by nights spent at the legendary club and venue the Haçienda. He also owns the Milk and Honey bar in Soho, and a self-styled “late-night dinky drinking den” on Brick Lane reassuringly called Danger of Death, as well as being a partner in three modern street-food festivals.

“Dealing with the local authorities – the council and the police – is a nightmare,” he says. “No matter how experienced you are, or how good your track record is, or what you’re proposing to do – it’s a battle and a challenge.” He gestures towards his new premises. “This was empty for 18 months. We’ve not kicked anyone out.” The surrounding area, he says, is full of “basement clubs for twentysomethings” – very different establishments from Rotorino, which opened just over year ago. “But we’ve had nothing but hassle trying to get a licence beyond 11pm. This is one of the most happening places on the planet, and I can’t open till midnight – even with the support of the police.”

‘Nightclubs are an engine of social mobility. They are almost a business school for Hackney’ … Dan Beaumont, owner of Dalston Superstore (pictured). Photograph: AIecsandra Raluca Dragoi for the Guardian

Nearby are two businesses co-owned by 37-year-old Dan Beaumont. Six years ago, he pioneered the expansion of local night-life, opening a bar called Dalston Superstore, and eventually adding a pizza outlet called Voodoo Ray’s (named after a much-loved 1988 acid house record by the Mancunian artist A Guy Called Gerald) and a club built into the same premises called the Dance Tunnel.

Like Downey, he bemoans Hackney council’s new Dalston “Special Policy Area”, which stretches along the A10 and was proposed “to help reduce late-night noise and anti-social behaviour.” Hackney council says it is based on the recognition that Dalston is “one of the most vibrant parts of the capital, with a great selection of restaurants, bars and clubs that attract people from across London and further afield” which “is great for the local economy and provides jobs. But it also means that we need to manage this growth to make sure that the number of pubs, clubs and restaurants, as well as the type of venue, does not have a negative impact on the local community in Dalston.”

When it comes to new applications or revisions of existing licences, the policy effectively means there will be no new places opening beyond midnight; 84% of respondents to a council consultation were against the idea, and even within the policy’s official catchment area only 40% were in favour, but it was passed regardless.

What this highlights, perhaps, is the fact that officialdom has yet to catch up with changes that first arrived more than 25 years ago. The acid house era began to energise the night-time economy and legions of small businesses in the late 1980s; a modern, internationalised urban culture now means that the idea of obediently going home at 11pm often looks hopelessly archaic. But too many people in power are still behaving as if British cities might somehow be returned to an imaginary time before all this happened.

Moreover, if bars and pubs are much misunderstood, nightclubs have an even worse time of it. “They’re still not seen as a culturally valid thing to do,” says Beaumont. “All the economic benefits of the late-night economy are well known, but there are cultural and social factors, too. Nightclubs are an engine of social mobility. They are almost a business school for Hackney. They are how you find out the basics of running your own thing, with low barriers to entry. And they are where you learn about people who are different from you: they are a really important expression of diversity.”

In a tiny booth by the bar, tonight’s DJ is riffling through her records. After a couple of drinks, you might imagine that you were in one of the groovier areas of Berlin, or Barcelona, or New York’s East Village. Only one figure does not quite fit the mood of bohemian languor: Alan Miller, sitting at a table close to the door, going through his messages, still trying to convince the right people that the British night-time is not the ghoulish void of some people’s paranoid fantasies, but a big part of the economic future.