We live in a golden age of sleeplessness. The buzz of the all-night streetlamps, the natter of 24-hour news anchors, the scrolling Niagaras of social media feeds have built a world that is hostile to sleep. Night is no longer clearly delineated from day. The bedroom is no longer a refuge from the office. The physical and psychic walls that once held back the tides of work and social interaction have failed. As the essayist Jonathan Crary put it, sleeplessness is the inevitable symptom of an era in which we are encouraged to be both unceasing consumers and unceasing creators.

To the wakeful, insomnia can feel like the loneliest affliction in the world. But an estimated third of British adults suffer from chronic insomnia, defined as having adequate opportunity but inadequate ability to sleep, for a period of at least six months. Insomniacs dutifully set aside a seven-or-so-hour stretch for rest. They make the bed. They draw the curtains. But when ear kisses pillow, they are suddenly wakeful. Many have sought help. Between 1993 and 2007, the number of people in the UK who visited their doctor complaining of insomnia nearly doubled, while NHS data shows, in the past decade, a tenfold increase in the number of prescriptions written for melatonin, the hormone that regulates sleep.

The effects of insomnia can be ruinous. In his recent bestseller, Why We Sleep, the neuroscientist Matthew Walker wrote: “The decimation of sleep throughout industrialised nations is having a catastrophic impact on our health, our life expectancy, our safety, our productivity and the education of our children.” A 2016 report by the Centers for Disease Control and Prevention, claims that insomnia increases the risk of heart attack, cancer and obesity. Insomniacs are far more likely than sound sleepers to suffer from chronic depression. Insomnia is related to all major psychiatric conditions, including suicide risk (although there is still a debate as to whether sleeplessness is the cause or the symptom). Each year, as many as 1.2m car crashes in the UScan be attributed to tired drivers.

None of this is news to the droopily Googling insomniac who, fearing obesity, heart disease, accident and poverty, is subjected to yet further sleep-skewering anxiety. Fearing their problem is untreatable, or that no doctor will take them seriously, many people who suffer from insomnia never seek medical advice. And in Britain, where doctors are hesitant to prescribe sleep drugs for longer than a week or two, who can blame the insomniac? There are a few NHS sleep clinics in the UK, where patients can be tested for the respiratory issues that often cause sleeplessness, but waiting lists are dispiritingly long. Moreover, for decades, inside the British medical establishment there has been only a glancing interest in insomnia, a specialism that one consultant refers to as the “Cinderella of medicine”.

“We have very little at our disposal,” Clare Aitchison, a GP with a practice in Norwich, told me. “In a 10-minute consultation it’s impossible to teach people to break bad habits.” With so few options, doctors resort to advisory cliches. Take a hot shower before bed. Eat a banana. Switch off your phone. Read a book. Masturbate. These titbits often have some basis in science or logic. But when the insomniac has tried them all (sometimes simultaneously) where do they turn?

There is, it turns out, a London clinic that has achieved remarkable results. Founded in 2009 by Hugh Selsick, a South African psychiatrist, the Insomnia Clinic in Bloomsbury has revolutionised treatment for sleeplessness in the UK. As Britain’s only dedicated insomnia facility, more than 1,000 patients have passed through the clinic at a rate that has quickened to, in 2018, 120 new casesa month. According to the clinic’s figures, 80% of patients report major improvements, while almost half claim to have been fully cured. This success has earned the clinic an enviable reputation and a waiting list to match; patients can wait two years for a consultation.

At the root of Selsick’s approach is a revolutionary assertion that has led to a new approach to treatment, quite unlike the old wives’ tales with which, in the absence of a coherent medical solution, every insomniac will be familiar. Where, for decades, insomnia has been treated as a symptom of another issue (if indeed it has been treated at all) Selsick contends that insomnia is not merely a symptom, but a disorder in its own right. This remains an unorthodox view. Yet, for Selsick’s patients, the approach does more than fix a category error: it provides a life-changing validation, a route out of helplessness, a way of getting to sleep.

I have come to hate my bedroom. What should be a place of rest and, in a good month, the odd romantic scuffle, has become a psychic battleground. Since I turned 18, the process of drifting off has become ever more easily torn. The pops and creaks of the settling house are enough to pull my wary brain from its slow descent. The sound of a lorry, or an orgasmic fox, can keep me tossing fretfully till 3am, the hour at which, as Ray Bradbury put it, we insomniacs glumly watch as “the moon rolls by ... with its idiot face”.

In the nagging light of the alarm clock, emotions grow heightened. The slightest wiggle, tut or guff from a bed partner is enough to arouse fury, as I’m whooshed back to a state of bounding wakefulness. The paradoxical exasperation of the insomniac is this: the more you try to sleep, the more you fail. So here I must lie, tipping from fury to dismay, contemplating the various ways in which the coming day is fucked.

It is impossible to explain to the sound sleeper what it is like to not sleep. Still, the writers and artists try. “Night is always a giant,” wrote Vladimir Nabokov, of the foreboding sense of peril he felt on entering his bedroom. (One of Nabokov’s insomniac characters wished for a third side after trying and failing to fall asleep on the two he had). Chuck Palahniuk, whose novel Fight Club was inspired by insomnia, would have to imagine picking and losing fights in order to drift off. F Scott Fitzgerald, not a writer prone to hyperbole, described insomnia with sullen childishness as “the worst thing in the world”.

Over the years I have developed rituals and incantations: the solemn depositing of the phone in a separate room, the scorching shower, the banana nightcap. As the fear of sleeplessness builds over weeks and months, obsessive, quasi-superstitious behaviours become established. Vincent van Gogh would pour a turpentine-like liquid on his mattress, a decanting intended to cast the spell of sleep. WC Fields claimed he could only fall asleep to the sound of rainfall, and his dutiful lover Carlotta Monti would spray water from the garden hose against the bedroom window till he dropped off (today, a range of apps can provide similar soothing soundscapes).

These eccentricities have, perhaps, enabled the rest of the world to view insomnia as a minor affliction. As well as feeling scorned, the insomniac comes to develop a sense of shame. Sleep is the most natural thing in the world; to fail makes the sufferer somehow unnatural. So it was with panda eyes and an anxious mind that I slipped through the front door of the Royal London Hospital for Integrated Medicine in Great Ormond Street, London, to meet the doyen of insomniacs.

Hugh Selsick cannot be absolutely certain, but he estimates that he has met more insomnia sufferers than any other person in Britain. Yet, when he enters the waiting room for his insomnia clinic, he has no idea which one of the expectant faces is his patient. Most long-term insomniacs show none of the telltale physical signs of fatigue. It’s a hidden, private affliction.

Selsick places extraordinary importance on this initial meeting with a new patient. He knows that they may have been suffering from insomnia for decades, a period over which they have seen multiple family doctors, who have time and again given them the kind of advice you might give a fretful child: take a warm bath or a glass of milk before bed. When he sits down with the patient for the first time, Selsick’s primary goal is simply to let them know, perhaps for the first time in their lives, that someone is about to take them seriously.

“For years nobody has understood what this person is going through,” he told me, as we sat in his narrow office. “Then suddenly they are sitting in front of someone who says, ‘Yes, I can see this is a problem, and yes, we can treat this.’” Some patients well up. Others hold their heads in their hands, in shocked relief. Whatever the reaction, Selsick, who is soft-spoken, kind-eyed and bald as an acorn, said that, in that moment, a bond of trust is established that is stronger than any other he has known in his career as a psychiatric doctor.

In this, our first meeting, I felt something of that emotional intimacy. Through shame, or a worry that he might think I was trying to jump the waiting list, I had not mentioned my own struggles with sleeplessness. His kind manner, and open acknowledgment of the pervasive awfulness of insomnia, was both comforting and thrilling.

Still, the insomnia clinic’s reputation has not been made on bedside manner alone. Selsick has designed a five-week programme that combines cognitive behavioural therapy (CBT), designed to break a person’s negative associations with their bedroom, and the entire business of drifting off, with something Selsick terms “sleep efficiency training”, a calibrated reduction of the amount of time the patient spends in bed.

Today, Selsick and another consultant run the clinic with support from a GP who works one day a week and an associate specialist psychiatrist, who is supported by a trainee. Patients travel from around the country to visit, and about 80 patients visit the clinic’s weekly group classes. “We are constantly expanding, but we are still battling to meet the demand,” Selsick said.

Hugh Selsick (right) with Andrew Eaton, a clinical scientist, at the Insomnia Clinic. Photograph: Sarah Lee/The Guardian

How is a central London clinic successfully treating an illness that, for decades, medicine has failed to adequately address? The answer appears to be rooted in Selsick’s belief that insomnia is not merely a symptom of another, higher-order condition. For decades doctors would treat the primary condition – diabetes, cardiovascular disease, respiratory problems – expecting that fixing this would help the patient to sleep. This approach would often fail because, as one study puts it, insomnia is maintained by the “behaviours, cognitions and associations that patients adopt as they attempt to cope with poor sleep but that end up backfiring”.

Selsick believes that only by treating insomnia as a psychiatric disorder, with degrees of severity that range from mild to chronic, can the health service begin to develop and prescribe appropriate treatments. It is a pioneering attitude that is motivated not only by scientific curiosity, but by personal experience; Selsick knows insomnia’s debilitating effects first-hand.

Selsick became an insomniac in 1993, when he was 19 years old, staying on a kibbutz in the desert in Israel. It wasn’t just the heat that caused his sleeplessness; it was also the routine built around the heat. With temperatures reaching 40C, desert-dwelling people typically sleep from 11pm through to 3am, at which point they start work while it is still cool enough. At lunchtime, when the heat is at its most ferocious, they take a siesta. It was a custom that Selsick’s mind resisted; he would lie awake in the afternoons, feeling exhausted, but wired.

When he returned to South Africa to start his first year at university, studying medicine in Johannesburg, Selsick’s insomnia persisted and intensified. “It’s almost impossible to describe what it’s like, to someone who hasn’t had it,” he told me. One day on campus he saw an advert posted on a wall requesting volunteers for a sleep study. Selsick enrolled in the hope that he might discover what was happening to him.

The study hoped to find what effect, if any, calorie intake had on a person’s ability to fall asleep. Each experiment lasted for four days, during which Selsick and the other volunteers would stay overnight at the sleep clinic, one monitor strapped to their head, another, to monitor core body temperature, inserted into their rectum. Volunteers were kept to a specific diet. One week they would fast for 24 hours; the next eat three times their typical calorie intake. Then they were monitored to see what effect the food had on their sleep. “It turned out to not make a difference,” he recalled.

Inspired by the professor who ran the course, Selsick began a postgraduate degree in physiology, based within the sleep clinic, where he studied the functions of REM sleep – a phase that occurs sporadically through the night, characterised by rapid eye movements – then carried out research on the impact of central heating on sleep patterns. (The ideal temperature in which to sleep is cooler than you might think: just 18C. This is one of the reasons why insomnia affects a disproportionate number of people in nursing homes, where round-the-clock heating makes it harder for the human body to cool down in readiness for sleep.)

At this time, the use of psychotherapy to treat insomnia was still in a relatively early phase. Selsick estimates that it was 2005 before therapists began to undergo insomnia training, in order to apply research findings. When Selsick came to London in the late 90s as a trainee doctor at the Royal College of Psychiatrists, his own insomnia had passed. Still, he was astonished to find a widespread lack of interest within psychiatry toward insomnia. “Ask any patient with psychiatric conditions what troubles them,” he said. “Sleep is almost always at the top of the list.” Selsick started a mailing list for any psychiatrists who were interested in sleep and held a conference at which members shared their findings. The group caught the attention of his supervisor, Charlotte Feinmann, a consultant psychiatrist at University College London Hospitals (UCLH) who, while Googling “insomnia”, recognised Selsick’s name in a search result. She sent him a text message asking whether he would be interested in founding an insomnia clinic at the hospital.

“At that time nobody was treating insomnia,” Selsick recalled. “Mental health units weren’t taking insomnia patients; sleep disorder centres were not treating insomnia, partly because they were being run by respiratory physicians screening for sleep apnoea, who didn’t have the relevant skills.” A patient who did not fit that box would be “bounced around the NHS”, Feinmann said. While staff across the health service were aware of the need, Selsick said, they knew that if they were to take insomnia referrals they would be inundated.

Selsick accepted Feinmann’s offer and, in November 2009, his first two patients walked into the clinic. He started small – one afternoon a week. “I had no idea what I was doing,” he recalled. Indeed, in its early months, Selsick’s consultancy offered little more than routine advice on basic sleep hygiene, such as limiting caffeine intake (“not effective”) and some general tinkering with the dosage of whatever medication the patient was already taking (“not very effective”).

Then, a few months later, Selsick began to explore CBT. For those who suffer from insomnia, the bedroom is so strongly associated with wakefulness that the mere act of going to bed wakes the patient up, in much the same way that walking into a dentist’s office makes you anxious. CBT, which at the time was just starting to be used to treat insomnia in North America, works to change patients’ automatic, often unconscious, association of the bedroom with wakefulness and replace it with bedroom and sleep. “Immediately,” Selsick said, “our results were enormously better.”

Not everyone was convinced by the new programme. Selsick’s clinic is situated within the Royal London Hospital for Integrated Medicine, formerly known as the Royal London Homoeopathic hospital, a controversial centre for alternative treatments. The pharmacologist David Colquhoun once described the hospital as a “great national embarrassment”. Selsick believes that this association caused some GPs to not refer their insomniac patients. “When we explain that we are a psychiatry-led service that practices evidence-based medicine, those issues usually melt away,” he explained.

For those who make it through the door, Selsick provides an initial assessment in an attempt to find out what, of a constellation of different possibilities, is causing the sleeplessness. He screens for sleep disorders such as restless leg syndrome, which affects 2%-10% of people. Like other sleep clinics, he screens for sleep apnoea and other respiratory problems. But this is merely the first step in the process. Once these possible causes have been ruled out, Selsick asks a long list of questions, both practical (“What time do you go to bed?” “How long does it take to fall asleep?”) and probing (“What was occurring in your life when you first started suffering from insomnia?”).

Ideally, the patient’s answers establish a pattern, which can lead to a diagnosis. Sometimes that diagnosis is narcolepsy, nocturnal epilepsy or sleepwalking – one of the scores of conditions that can lead to sleeplessness. In other cases it is, simply, psychiatric insomnia.

When she was 13 years old, Zehavah Handler took a pen and scrawled a dot on her bedroom wall in north-west London. Lying on the bed, she could just pick out the mark in the milky glow of her nightlight. There, as the house settled around her, she would challenge herself to stare at the dot for as long as possible without blinking. The game became a ritual and, eventually, became the only way, she believed, that she could fall asleep – although it was often 4am before she finally drifted off.

In adulthood Handler, now 40 and a mother of four, still suffered from insomnia. She would wake at 7am to drop her children off at school, then lie down on the carpet of her bedroom. There she would watch the ceiling till mid-afternoon, her heart palpitating with exhaustion, when she would leave for the school pickup. After feeding and bathing the children, Handler would retire to her own bed. She would lie in bed for 12 hours, only sleeping for an hour or so before the dawn broke and the day’s grim routine began again.

When she began to experience memory loss and irritability, Handler visited her GP. After an 18-month wait, she entered Selsick’s office. “It was the first time I met a professional who acknowledged the problem and was truly empathetic.” Handler was admitted to UCLH’s clinic to be monitored overnight for sleep apnoea. On arrival, Handler found the consultant in charge that night “extremely dismissive” of Selsick’s clinic. She spent the first night in a nest of wires, like an android recharging its batteries, lying awake worried about whether or not the machines would know that she was just pretending to be asleep. Nevertheless, the results came back clear: she had no breathing issues, no twitching muscles. Selsick concluded that Handler was one of his many patients for whom insomnia is not a symptom of some other disorder, but the disorder itself.

In May 2016, Handler joined Selsick’s five-week course, along with nine other anxious patients. The programme takes place in a small room in the bowels of the hospital. Handler remembers that none of her fellow patients talked and few made eye contact, paralysed by the secret shame of the insomniac. “Everybody was very self-conscious,” she recalled. “We were wondering: ‘How’s it going to work? How much will we have to reveal of ourselves?”

A patient at the sleep clinic. Photograph: Sarah Lee/The Guardian

“The first thing I do [on the programme],” Selsick told me, “is dispel the myth that there’s a certain number of hours you are supposed to get. It is ingrained in us as if it’s absolute gospel that you are supposed to sleep eight hours a night. It’s not true.” Just like there is variation in shoe size, Selsick says, there is variation in the amount of sleep an individual requires. “Some people need six and a half, some people need nine and a half. It doesn’t make anyone abnormal.”

To figure out how much sleep they need, each attendee is told to start a sleep diary, recording what time they go to bed, what time they get up, how long it took to fall asleep and how many times they wake during the night. Next Selsick undoes the idea that a person should have a set bedtime. Typically, insomniacs will go to bed earlier or stay in bed longer in order to increase their sleep opportunity. The logic appears sound – if I’m not getting enough sleep, I should spend longer in bed to give myself more opportunity to sleep – but the anxiety invariably exacerbates the problem. Instead, patients are told to set a hard-and-fast time to wake up. “We tell them to always get up at the same time every day, regardless of how much they’ve slept, what time they’ve gone to bed, or what they’ve got to do that day.”

There must never be any lie-ins, and never any naps (chewing gum, Selsick says, keeps napping at bay). The theory is that if you get up at the same time every morning, you begin to feel sleepy at the same time every night, and, over the weeks, your bedtime will naturally become consistent. “We compress their time in bed down so that their sleep is more compact and tighter,” explained Selsick. A patient might start with a goal of six hours’ sleep. If they need to be up for work at 7am, this means they are forbidden from entering the bedroom till 1am. “That’s now your earliest permitted bedtime.”

Once a patient finds they are asleep for 90% of the time that they are in bed, they move that earliest bedtime forward by 15 minutes at a time. This behavioural technique is termed sleep efficiency, and despite its disarming simplicity, patients report astonishing results. “It was very hard going,” said Laurell Turner, a medical student who completed the programme in 2016. “By the end of the course I was exhausted. But despite my scepticism, the results were immediate.”

Selsick worked to break the negative associations Handler had with her bedroom. When insomniacs go to bed they often feel afraid of having to lie there frustrated and increasingly irritated. After a while the simple act of going to bed begins to wake an insomniac up. The bedroom becomes a trigger toward alertness, even fear. To counter this, Selsick urges patients to leave the bedroom after just 15 minutes if they are not yet asleep. All activity apart from sex and sleep is barred from the bedroom. Patients are even told to change their clothes in another room.

“Before, I would go to sleep in the afternoon and be in my room for 12 hours,” said Handler. I would make all my phone calls there, work on my laptop there, eat and watch TV in bed. That’s gone. All gone. I say goodbye to my room at about 7.20am and I don’t see it again until 1.30am when I go to bed.” The technique often feels counterintuitive; on the initial nights that a patient shuffles between bedroom and lounge every 15 minutes, they often sleep worse. “It’s extraordinarily hard to do,” she said. But after about five weeks, the negative psychological association of the bedroom with wakefulness has been broken, and replaced with new, positive connections. Selsick claims that, using these techniques alongside the moderation of stimulants such as caffeine, eight out of 10 patients get better, and half of these go into, he said, “complete remission”.

Studies show that CBT is the most effective long-term treatment for insomnia. But in order for it to be effective, it requires the patient to establish and maintain a stable routine. For patients who regularly cross time-zones, who often stay in unfamiliar hotel beds or who are unable to form a nightly ritual because of work, Selsick’s plan presents an impossible target. These patients want not a timetable to which they must stick, but a pill they can swallow.

Surprisingly perhaps, for a doctor who strongly advocates the use of CBT in treating insomnia, Selsick believes that sleeping pills should be far more widely prescribed in the UK. “There’s an incredible conservatism in the British medical establishment about prescribing for sleep,” he said. Much of this anxiety is focused on the addictive qualities of benzodiazepines. According to the neuroscientist Matthew Walker, sleeping pills do not provide “natural sleep,” can “damage health” and “increase the risk of life-threatening diseases”.

“The drugs, like any drug, are not without risk,” said Selsick. “But having untreated insomnia also comes with risks.” Selsick has met patients who, because of insomnia, have been forced to leave work and abandon careers. “I’ve had patients where it’s destroyed their marriages, where they have lost access to their kids because they are so tired that they can’t look after them properly.” Nevertheless, Selsick perceives a blanket policy among doctors whereby they refuse to medicate for sleep. This policy, he says, does patients a disservice. “Yes, CBT should be your first port of call prior to medication. But most places in the country do not have access to CBT. And not everyone who does CBT for insomnia gets better.”

The arrival of any epidemic brings commercial opportunities. In 2006, the manufacturer of the non-benzodiazepine sleeping pill Ambien estimated that the drug had been taken 12bn times worldwide and is worth $2bn a year in American sales. Pharmaceutical companies hoping to replicate this success are locked in a race to design a new sleeping pill without side-effects. The discovery, in 1998, of orexin, a hormone that essentially works as the brain’s alarm clock, turned the long march to develop a new kind of sleeping pill into a sprint.

For the past 15 years Jean-Paul Clozel – a cardiologist turned pharmacologist who in 1997, co-founded the Swiss biotech firm Actelion with his wife, Martine – has been leading the development of what he claims to be the side-effect-free sleeping pill. “Most sleeping pills are benzodiazepines,” says Clozel. “They induce something that feels like sleep, but in reality, it’s closer to anaesthetic sedation.” (Benzodiazepines are often used by anaesthetists.) Clozel’s pill, which he hopes to take to market in 2020, and which goes by the generic name Nemorexant, works differently. It limits the production of orexin, the hormone that keeps insomniacs awake, or causes them to wake up at the mildest provocation.

Nemorexant is not the first sleeping pill to target orexin. Since August 2014, more than a decade after work began on developing the drug, American doctors could prescribe Belsomra, also known as Suvorexant, which targets the same hormone. Within a month of its release, US doctors were writing an average of 4,000 Belsomra prescriptions a week. But the drug is not without risks. An FDA report on the safety of Belsomra, which is closely related to Clozel’s drug, quoted one patient who “woke up several times and felt unable to move her arms and legs and unable to speak”.

Nevertheless, in a country that is seemingly years away from rolling-out nationwide CBT programmes to treat insomnia, Selsick welcomes the chance to prescribe Neomorexint. “As it acts through a completely different pathway than other hypnotics, it would be nice to have it for those patients who have not responded to the standard treatments.”

In the meantime, Britain remains both ill-equipped and, seemingly, unwilling to deal with the growing insomnia epidemic. As victims of the woefully ignored Cinderella of medicine, sufferers struggle to stick with anything that claims to be a cure, leaving us stuck in folklore, with its colourful but contradictory advice. No sleeping pills are licensed for long-term usage, and apart from Selsick’s clinic, only a handful of private psychology services offer CBT to treat insomnia.

A plan to open an insomnia clinic at the sleep disorders centre at Guy’s hospital was scrapped due to fears that the demand would be too great. “They were concerned that demand would be so massive they would never meet their waiting list targets, which would result in being financially penalised,” Selsick said. This has resulted in the perverse situation where the more demand that exists to treat insomnia, the less likely it is to be met.

In May, to help ease the pressure on his oversubscribed clinic, Selsick commissioned the first insomnia training programme for GPs, an attempt to equip doctors to run, within their local surgeries, CBT sessions similar to those held at the clinic. Eventually, he hopes, this will mean that only the most extreme cases will need to be referred to him. Selsick hopes to run these courses, which are also open to nurses, psychologists, occupational therapists and mental health workers, twice a year and, in doing so, speed up the NHS’s capacity to deal with insomnia at a national scale.

Selsick’s clinic is the only surgery to consistently see a significant volume of patients. And consistency is, he said, the key. “The treatment is not rocket science,” he said. “It really isn’t. Our job, primarily, as therapists, is not so much to tell people what to do – because we could just give it to them as a handout – but to convince them to do it for long enough for it to work.”

For patients who successfully complete Selsick’s programme, the ability to sleep well changes life at an elemental level. To begin sleeping again is to feel as though you are being realigned with the universe and its guiding, imperceptible rhythms. “I am happier,” Handler told me, of her new life, post-insomnia. “My relationships have improved. I have more patience. I’m no longer walking in a permanent fog. I am available.”

There are occasional relapses, Handler said, usually brought on by a change in routine – a holiday away, Christmas – but by waking at a set time, leaving the bedroom after 15 minutes if she remains awake and re-implementing all of the rituals she learned at the Insomnia Clinic, it only takes a few nights to re-establish the routine.

The effect has been so transformative that Handler has decided to close her tourism business and, with Selsick’s support, retrain as a sleep counsellor. So great is her relief at learning how to sleep again that Handler wants to dedicate her life to helping others do the same; she plans to open her own insomnia clinic next year.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.