Every night, about a third of adults have problems falling or staying asleep that aren't related to a persistent sleep disorder. As they lie in bed, many are caught in the classic paradox of insomnia: wanting sleep so badly that they can't get it. "The condition of sleep is profoundly contradictory," notes Emily Martin, a professor at New York University who has studied insomnia. "It is a precious good … but it is a good like none other, because to obtain it one must seemingly give up the imperative to have it."

For doctors, insomnia presents a chicken or egg problem. Is sleeplessness a result of another condition such as depression, or is the insomnia the root of the other problem? One report by the US National Institute of Mental Health found that depression rates were 40 times higher for patients with insomnia than those without sleep problems. Mental health experts increasingly view depression or anxiety as an effect, rather than a cause, of insomnia. Taking care of insomnia may therefore calm other aspects of a patient's life.

And yet insomnia is a unique and difficult condition to treat because it is self-inflicted. The cause is often the brain's refusal to give up its unequalled ability to think about itself, a metaphenomenon that Harvard professor Daniel M Wegner has called "the ironic process of mental control". To illustrate this concept, imagine someone telling you that you will be judged on how quickly you can relax. Your initial reaction most likely is to tighten up. After he posed that challenge to research subjects, Wegner found that the average person becomes anxious as his or her mind constantly monitors its progress toward its goal, caught up in the second-by-second process of self-assessment.

In the same way, sleep becomes more elusive as a person's sleep needs become more urgent. This problem compounds itself each night, leading to a state of chronic insomnia.

Treating insomnia isn't easy. Part of the reason is that science, as a whole, has a fuzzy definition of what constitutes the disorder. One night of bad sleep because of a blaring car alarm or an upcoming stressful day at work doesn't classify as insomnia. Instead, it is generally thought of as a string of otherwise peaceful nights during which a patient can't fall asleep when he or she wants to. The US National Institutes of Health (NIH) identifies the condition as "difficulty getting or staying asleep, or having non-refreshing sleep for at least one month". The classic form has no known cause, yet is widespread. About one in 10 people in the US suffer from it during their lifetime.

There is no medical test that proves whether someone is suffering from a temporary bout of sleepless nights or a more serious disorder. Some patients go to sleep labs and undergo tests to rule out conditions such as sleep apnea, but knowing what they don't have offers little help in treating what they do.

Instead, doctors rely on self-reports from patients, which can be maddeningly vague, a result of the difficulty that we have with accurately noting how many hours we truly spent sleeping on any given night. Patients who have spent a night in a sleep lab, for instance, often complain that it took them more than an hour to fall asleep when a chart of their brain waves shows they were asleep within 10 minutes. Problems of self-reporting aren't limited to judging how long it took to get to sleep. Some patients wake up in labs claiming that they didn't sleep at all during the night, despite hours of video and brain wave evidence to the contrary.

It is a part of the paradox that sleep presents to a conscious mind. We can't easily judge the time that we are asleep because that time feels like an absence, a break from the demands of thought and awareness. The times that we do remember are those that we wish we couldn't: staring at the clock in the middle of the night, turning the pillow over desperately hoping that the other side is cooler, kicking the covers off or pulling them up close. Those experiences, even if they last only three minutes, often become exaggerated in our minds and overshadow the hours that we spent sleeping peacefully, simply because we remember them.

When insomnia starts to interfere with the routines of normal life, many people turn to pharmaceuticals.

Medicines that help someone fall asleep, stay asleep, or be comfortable in between accounted for $30bn (£18.5bn) in 2010 in the US alone, which is more than what people around the world spend each year going to the movies.

The sleeping pill market changed in 1993 when a French company now known as Sanofi introduced a drug called Ambien, also known by its generic name zolpidem. Ambien appeared safe enough for many doctors to break their long-standing refusal to prescribe a medication for run-of-the-mill insomnia. It quickly dominated the sleeping-pill market and rang up more than $1bn in sales a year. At one time, Ambien accounted for eight out of every 10 sleeping aids prescribed in the US, a near monopoly enjoyed by few other drugs in history.

But here's the twist. A number of studies have shown that Ambien and other shorter-acting benzodiazepines, sometimes known as Z-drugs, such as Zimovane offer no significant improvement in the quality of sleep that a person gets. They give only a tiny bit more in the quantity department, too. In one study financed by the NIH, patients taking popular prescription sleeping pills fell asleep just 12 minutes faster than those given a sugar pill, and slept for a grand total of only 11 minutes longer throughout the night.

If popular sleeping pills don't offer a major boost in sleep time or quality, then why do so many people take them? Part of the answer is the well-known placebo effect. Taking any pill, even one filled with sugar, can give some measure of comfort.

But sleeping pills do something more than that. Drugs like Ambien have the curious effect of causing what is known as anterograde amnesia. The drug makes it temporarily harder for the brain to form new short-term memories. This explains why those who take a pill may toss and turn in the middle of the night but say the next day that they slept soundly. Their brains simply weren't recording all those fleeting minutes of wakefulness, allowing them to face each morning with a clean slate, unaware of anything that happened over the last six or seven hours. Some sleep doctors argue that this isn't such a bad thing. "If you forget how long you lay in bed tossing and turning, in some ways that's just as good as sleeping," one physician who worked with pharmaceutical companies told the New York Times, voicing what is a widely held opinion among the sleep doctors I spoke with.

Serious problems can arise, however, when people taking a drug like Ambien don't actually stay in bed. Some have complained of waking up the next day and finding sweet wrappers in their beds, lit stoves in their kitchens, and bite marks on the pizzas in their freezers. Others have discovered broken wrists that came from falling while sleepwalking, or picked up their cell phones and seen a list of calls that they have no memory of making.

Not long after a member of the Kennedy family blamed a car accident on the effects of Ambien, the US Food and Drug Administration issued new rules requiring pharmacists to explain the risk that taking certain sleeping pills could lead to things like sleep-eating, sleep-walking, or sleep-driving.

Those warnings have done little to dent the popularity of sleeping pills, especially since the most popular one is cheaper than ever. Ambien went off-patent a few months before the FDA issued its new requirements. The number of patients filling a prescription for them remained steady. Many people who take sleeping pills find that their sleep quality reverts to its previous, poor state the night they decide to go without medication, a vicious cycle that increases their dependency on a drug approved only for short-term use. Facing a night of sleep without backup produces the same form of stress that originally caused the insomnia cycle to begin.

Yet there is a way to treat insomnia without setting patients up for a letdown as soon as the prescription runs out. Charles Morin is a professor of psychology at Université Laval in Quebec. For more than 10 years, he has conducted studies into whether modifying behaviour can be as effective at treating insomnia as taking medication. His research focuses on cognitive behavioural therapy (CBT), a treatment that psychologists often use when working with patients suffering from depression, anxiety disorders or phobias. The therapy has two parts. Patients are taught to identify and challenge worrying thoughts when they come up. At the same time, they are asked to record all of their daily actions so that they can visualise the outcome of their choices.

When used as a treatment for insomnia, this form of therapy often focuses on helping patients let go of the fear that getting inadequate sleep will make them useless the next day. It works to counter another irony of insomnia: Morin found that people who can't sleep often expect more out of it than people who can.

Patients with insomnia tend to think that one night of poor sleep leads to health problems or has a severe impact on their mood the next day, a mental pressure-cooker that leaves them fretting that every second they are awake in the middle of the night is another grain of salt in the wound. In the inverted logic of the condition, sleep is extremely important to someone with insomnia. Therefore, the person with insomnia can't get sleep.

In a study in 1999, Morin recruited 78 test subjects who were over the age of 55 and had dealt with chronic insomnia for at least 15 years. He separated his subjects into four groups. One group was given a sleeping pill called Restoril (temazepam), a benzodiazepine sedative often prescribed for short-term insomnia. Another group was treated with CBT techniques that focused on improving their expectations and habits when it came to sleep. The members of this group were prompted to keep a sleep diary and talk to a counsellor, as well as carry out other actions. The third group was given a placebo, and the fourth was treated with a combination of Restoril and the therapy techniques.

The experiment lasted for eight weeks. Morin then interviewed all of the subjects about their new sleeping habits and the quality of their sleep each night. Patients who had taken the pill reported the most dramatic improvements in the first days of the study, sleeping through the night without spending any of the lonely hours awake they had come to expect. Subjects who were treated with CBT began to report similar results in sleep quality a few days later. Over the short term, sleeping pills had a slight edge in smoothing down the rough edges of insomnia.

Then after two years, he contacted all his subjects and asked them about their sleeping habits again. It was a novel approach to investigating a disorder that often appears solved as soon as a patient sleeps normally for a few nights.

Morin wanted to determine whether sleeping pills or therapy would do a better job of reshaping the underlying causes of persistent insomnia. Subjects who had taken the sleeping pills during the study told him that their insomnia returned as soon as the drugs ran out. But most of those who went through therapy maintained the improvements they had reported in the initial study. Lowering patients' expectations of sleep and helping them recognise what contributed to their insomnia combined to be more powerful over the long term than medication. "In the short run, medication is helpful," Morin told the New York Times. "But in the long run, people need to change their actual sleep habits — that's what CBT helps them do."

Therapy is also helpful at breaking a person's reliance, either real or imagined, on sleeping pills. In a 2004 study, Morin found that nine of every 10 subjects who combined a gradual reduction in their medication with CBT were drug-free after seven weeks. Only half of those who tried to stop using the pills by reducing dosage alone were as successful. Further tests revealed that subjects who relied on therapy experienced better sleep quality as well, with longer amounts of time in deep sleep and REM sleep. A separate study the same year found that one out of two subjects who began a cognitive behavioural treatment plan no longer felt the need to take sleeping pills. The results from these and other CBT studies have been compelling enough for organisations ranging from the National Institutes of Health to the NHS to recommend therapy as a technique for treating insomnia.

Yet some people with insomnia may never respond to therapy like this, simply because their sleeplessness isn't a reflection of the mind putting pressure on itself. Instead, it may be due to nothing more than age. As we get older, the structure of our sleep undergoes subtle changes. The amount of time that adults spend each night in REM sleep begins to decline at around the age of 40. At that age, the brain begins a process of readjusting its sleep pattern and devoting more time to the lighter stages of sleep. Soon the barking dog that someone was able to sleep through at the age of 25 is a nuisance that makes sleep impossible. These changes, a decade in the making, often become more apparent once someone turns 50. By the time a person reaches 65, he or she often settles into a pattern marked by falling asleep around nine o'clock at night and waking up at three or four in the morning.

What many older adults call insomnia may in fact be an ancient survival mechanism. Carol Worthman, an anthropologist at Emory University in Atlanta, has argued that the modern comforts of silence, deep foam mattresses, and climate control have given us the expectation that sleep should always come easily. The wiring of our brains hasn't caught up with the comforts of our bedrooms, however. Early humans were at their most defenceless when they laid down on the ground for several hours in the middle of the night.

Sleeping patterns that change as we age show that our brains expect us to be living and sleeping in a group, Worthman says. To illustrate this idea, she noted that the three basic stages of adulthood – teenage, middle age, old age – have drastically different sleep structures. Teenagers going through puberty find it impossible to fall asleep early and would naturally sleep past 10 in the morning if given the choice. Their grandparents often fall asleep early in the night, but then find that they can't stay that way for more than three or four hours at a time.

Middle-aged adults typically fall between these two extremes, content to fall asleep early when circumstances allow it, yet able to pull an all-nighter when a work project calls for it. These overlapping shifts could be a way to ensure that someone in the family is always awake and keeping watch, or at least close to it. In this ancient system, it makes sense that older adults who are unable to move as fast as the rest of the family are naturally jumpy, never staying in deep sleep for long, simply because they were the most vulnerable to the unknown. Those survival instincts are of little help when life takes place in a bungalow in the suburbs.

This is an edited extract