When I was a hormone-addled adolescent in the late 1960s and early ’70s, I would often look up at a poster of Sigmund Freud on my brother’s bedroom wall. The title on the portrait – something like ‘Freud: explorer of the unconscious and discoverer of the meaning of dreams’ – depicted a hero of intellectual freedom and creative thought. When you looked at it closely, the portrait seemed to writhe and come alive. In the drug-fuelled style of those decades of ongoing sexual revolution, the artist had depicted the nose as an erect penis, the cheeks as a female behind, and the eyes as female breasts. One side of the face was a voluptuous female whose legs wrapped around the body of a muscular male on the other side of the face and, of course, both heads were thrown back in dramatised ecstasy. I recall some of my brother’s stoned friends gazing at the portrait with bewildered looks on their faces, apparently unsure if the writhing torsos they saw were really there or not.

Right from the start, I saw Freud as a kind of secular saint because he was willing to take an unbiased look at himself through the raw material of his dreams. If he found in those dreams a mass of broiling sexual impulses, so be it. Those impulses had to be accepted, understood and explained within a larger picture of the human mind.

It was on the night of 23 July 1895 that Freud had his famous ‘dream of Irma’s injection’ – the first he analysed. In the dream, Freud met Irma, a young widow and ailing patient under his care, at a party – ‘A large hall – numerous guests.’ He took Irma to the side, reproaching her for not taking his advice. ‘If you still get pains, it’s really only your fault,’ he said. To which Irma replied: ‘If you only knew what pains I’ve got now in my throat and stomach and abdomen – it’s choking me.’ Freud was alarmed. Irma looked pale and puffy. When Freud looked down her throat, he found a big patch of white. And he knew the origin of the infection: not long before, his friend Otto had given her an injection of trimethylamine. ‘I saw before me the formula for this printed in heavy type … Injections of this sort ought not to be given so thoughtlessly … And probably the syringe had not been clean,’ Freud later wrote in his masterwork, The Interpretation of Dreams (1900).

He thought his dream represented wish-fulfilment: by blaming Irma’s treatment failure on others, he could relieve himself of feeling guilty. But how was it a sexual wish? Freud noted that while the party was apparently a birthday celebration for his wife, his attention had focused on Irma, who reminded him of a young widow he desired to treat instead; his friends, including Otto, were portrayed as competitors. Freud’s commentary also revealed that the reference to trimethylamine came from his close friend, Wilhelm Fliess, who had labelled the substance a ‘product of sexual metabolism’ found in semen.

To Freud, ‘Irma’s injection’ represented clear support for his theory that dreams amounted to sexual wish-fulfilment. But critics piled on. There was the obvious fact that his theory seemed to be based on idiosyncratic associations, quite literally open to endless interpretation.

By the time I started my own dream research as a graduate student, Freud’s theories had been relegated to the dustbin of history – not even worthy of the name science, as the Harvard psychiatrist and dream researcher J Allan Hobson said. The derision started full-bore after 1953, when the University of Chicago physiologist Nathaniel Kleitman and his student Eugene Aserinsky discovered rapid eye movement sleep, or REM, as the backdrop for vivid dreams. In REM, which occurs every 90 minutes or so, the eyes dart back and forth under closed lids while muscles become paralysed and body temperature drops, all as the dream unfolds. To many, the clear physiological nature of the phenomenon seemed to upend Freud’s sexual theory, with its gritty, emotional, dirty, smelly and earthy cast. Instead, there emerged a new, cognitive approach to dream content. Those theories treated dreams as disembodied, ethereal and pristine products of an otherwise neutral information-processing machine we call the brain. Researchers such as Hobson saw dreams as nothing more than fanciful, ad hoc interpretations of random neural impulses rippling up from the brainstem, the engine of REM.

Yet I never understood why the discovery of REM sleep meant Freud had to be wrong. To me, REM was another crucial piece of the dream puzzle – and could coexist easily alongside Freud’s theories that dreams had a deep unconscious meaning and purpose, overall. That purpose had to be rooted in evolution, I thought – one way or another, dreams helped us to survive.

I vividly recall the day in the late 1970s when I realised that dreams and their unconscious sexual meaning were part of a larger whole. I was 18 and working in my first real job, as an orderly at a major detox centre in the skid-row section of Boston. I and another orderly were given the task of delousing, showering and cleaning up an old alcoholic who had been picked up off the streets for a drying-out period. The old man was sitting under the shower letting the warm water slowly wash away the grime and dirt that covered his ravaged body. We threw the guy some soap and wash towels, and tried unsuccessfully to avoid smelling the stench that wafted off his rail-thin frame, when all of a sudden this emaciated, brittle old man jumped up, stared straight at us revealing a full erection and then lifted a massive metal table over his head, threw it against the wall and began wailing in ever louder sing-song tones a string of sexual expletives that left me and my colleague terrified that the man was crumbling, psychically, before our eyes. A nurse quickly arrived and told us he was going into DTs, or delirium tremens, often the result of alcohol withdrawal. She finally calmed him down with a shot in the arm. When she told us that the guy was essentially acting out a dream or nightmare, and that DTs emerged out of REM sleep (which I later discovered was only partly true), I knew I wanted to learn more. REM sleep not only produced those things (dreams) I had long been interested in (due to Freud), but it was also capable, apparently, of turning a ravaged old man into a raging bull.

As years passed and my youthful fascination turned into professional research, the connections between REM sleep, dreams and sex hit me again and again: like the skid-row alcoholic, men experience very prominent erections that begin with the REM period. Women experience clitoral engorgement with REM. And recent studies with functional magnetic resonance imaging (fMRI) reveal that the brain’s reward centres and circuits are highly active during REM sleep.

Hoping for more insight, I spent my time as a graduate student in neuroscience at Boston University collecting dream reports from anyone who would share them with me. Over the years, themes began to emerge. For example, I became utterly convinced that men and women were dreaming drastically different types of dreams, with sex as a common theme. Men were always on some kind of adventure or engaging in some kind of violent war or dramatic struggle with other men, while women were typically talking animatedly with friends or other people they knew.

Ultimately I looked to dreambank.net, a collection of more than 20,000 dream reports collated by psychologists at the University of California at Santa Cruz. In one dream from the archives, for instance, a male college student explains that he was in a theatre restaurant with his date when she mentioned that a man on stage had previously insulted her and severely beaten her escort. ‘I climbed up on the stage and attacked him,’ the dreamer wrote. ‘He was about 25 and very formidable-looking but catching him by surprise I succeeded in knocking him down. The audience thought it was part of the performance and applauded.’

In their dreams, men engage in physical aggression against other men; women engage in verbal rejections and exclusions of other women

Now look at a dream from a female college student, drawn from the same group of subjects: ‘I dreamt that a friend of mine who graduated last year came back to the dorm for Stunt Night. Another friend took care of her and gave her my bed to sleep in. Meanwhile another girl whom I’m not too friendly with was engaged to a boy whom she was not very much in love with. He was very wealthy and her ring was so beautiful that she didn’t want to wear it around school. She was always putting her arms around me … a very affectionate girl. … Later I went downstairs and my friend who took care of the visitor and I proceeded to tell her about our affairs at school and our respective boyfriends.’

While both dreams included romantic targets, the male dreamer describes aggression against potential competitors while the female dreamer subtly denigrates her competitor, the girl who received a beautiful ring. After I became a professor at Boston University in the mid-1990s, I confirmed these observations in rigorous studies: men dream more often of other men than they do of women, while women dream equally often of men and women. In addition, men more often engage in physical aggression against other men in dreams, while women more often engage in non-physical forms of aggression, for instance verbal rejections and exclusions of others.

But did these reports support Freud’s claims that dreams were essentially all about sex? I thought so, and my reasoning went like this: men dream more about aggressive interactions with other men because in our evolutionary past that was the way to get access to fertile females. Similarly, women more often dream about engaging in verbal aggression because gossip against competitors was a reproductive strategy for females.

I spent two decades trying to prove this simple hypothesis correct.

In one line of inquiry, I found ample evidence in the literature that sex hormones surged with REM sleep and dreams. Prolactin, which enables mothers to produce milk and stimulates the testicles, rises rapidly as sleep sets in and peaks between 3am to 5am, when REM predominates. Release of prolactin can be blocked by sleep deprivation. Similarly, oxytocin, which is linked to bonding during sex, and testosterone, which is tied to the sex drive, both peak at about 4am, when REM rules. This all chimes with fMRI scans that show peak activation of the midbrain – especially the circuits involved in pleasure, drug addiction, and sex – occurs during REM sleep.

But proving a deep connection between dream sleep and sexual wish-fulfilment required more: I would have to correlate specific dreams with particular mating strategies in life. After searching for a reliable measure, I chose attachment orientation – a quality associated with sexual and emotional intimacy.

Current attachment theory suggests that people fall into a few broad categories: they might be happy in their relationship, making them securely attached to their partner. They might not be in a relationship, but desperately want one, or be in one, but unhappily, making them preoccupied and anxious. Or they might be dismissive about the importance of relationships, and thus, avoidant.

The anxious individual is passionate about getting into a relationship with a romantic target, and thus recalls more vivid, emotional dreams of intimacy

Now if dreaming somehow reflects our sexual wish-fulfilment, then dream recall, dream content and dream sharing should be relatively lower in those who are satisfied with their current attachment orientation (secure, dismissive, and avoidant) and relatively higher among those who want to change their status (the preoccupied/anxious group). To test this idea, my team at Boston University recruited hundreds of volunteers until we had enough in each attachment category. We asked them about their dreams, and coders who were blind to the purpose of the study painstakingly analysed them.

When we collated the results, we were startled by what we found. The anxious, preoccupied group was far more likely to recall dreams than the securely attached; they took less time to enter REM sleep and had many more dreams featuring aggression against competitors. But both the anxious and the securely attached recalled more dreams than avoidant participants. That is precisely the pattern one would predict if dream sleep were directly related to long-term sexual strategies. The anxious individual is passionately interested in getting into a relationship with a romantic target, and thus recalls more especially vivid and emotional dreams filled with content concerning intimacy. The avoidant individual, conversely, suppresses the subconscious call for sexual closeness as reflected in dreams.

Finally, in a crucial test of the idea that all we dream about is sex, we decided to compare aggression in REM dreams with non-REM (NREM) dreams – in other words, shallower sleep states. If it turned out that high levels of aggression occurred only during REM, then it would strengthen the case that ties sex to this physiological state and all that it implies – the hormone surges, the activation of the brain’s pleasure centres, and the stimulation of genitalia, to mention just three factors that stand out.

Initially, we had 15 subjects fitted with a ‘nightcap’ designed by sleep researcher Robert Stickgold and his team at Harvard Medical School. Our Harvard colleagues programmed the nightcap to beep at random, waking subjects up so that they could recall dreams, during REM, during NREM, and even during periods of daytime wakefulness. Coders then scored dream reports for various aspects of aggression. Results lent dramatic support to our ideas: social interactions between men and women, and between men and men, were more likely to be depicted in dreams than in wake reports. Aggressive social interactions initiated by the dreamer were three times more frequent in REM than in NREM dreams or in wake reports. And friendliness initiated by the dreamer was twice as prevalent during NREM as in REM sleep. Later analyses showed that when the dreamer was male, targets of aggression during REM dreams could reasonably be described as competitors for a mate.

More recently, with a grant from the US National Institutes of Health, we replicated the results in a well-equipped sleep lab, using electroencephalogram (EEG) technology to measure brain waves, adding rigour and precision to the finds. In that study, hundreds of carefully selected college-age volunteers came to our sleep lab for several nights. They were subjected to a schedule of carefully balanced awakenings so that every participant had had a chance to recall dreams from both REM and NREM sleep.

We confirmed the findings of the previous study, but learnt more as well. For instance, when dreams involved aggression, the dreamer was the aggressor in 58 per cent of REM dreams but only in 29 per cent of NREM dreams. When dreams involved friendly interactions, the dreamer was the befriender in 71 per cent of NREM dreams and in only 42 per cent of REM dreams. In both studies the REM–NREM differences were most marked for males, and the target of the aggressions were competitors relative to the dreamer.

Yet even if dreams are all about sex, how would that explain why we put ourselves at so much risk just to run the late-night reel? For REM sleep is profoundly dangerous: the major antigravity muscles of the body are inhibited or paralysed, and the thermoregulatory reflexes of the body are suspended, making it impossible to produce much internal heat. REM is also associated with intense autonomic nervous system (ANS) ‘storms’ or instabilities. The ANS is the system that regulates key physiological processes such as the heartbeat; instabilities during REM explain why there are so many heart attacks in the wee hours of the morning when sleep is rich in dreams.

In short, every 90 minutes during sleep, we enter this dangerous twilight zone called REM sleep. The reward centres of our brains are activated, and our sexual systems are turned on – yet our bodies are partially paralysed so we cannot move to take advantage of these activations. Instead, as key physiologic systems collapse, we are forced to watch these things we call dreams. Why would evolution impose this risky system on us? When we are paralysed, we are vulnerable to predation. When we are unable to generate internal heat, we are vulnerable to disease. When our ANS becomes erratic, we are vulnerable to heart attacks, and so forth. So whatever REM sleep and dreams are doing for us it must be quite important, given the dangers they usher in. But what could that extremely important function be?

It wasn’t until I came across Charles Darwin’s book The Descent of Man, and Selection in Relation to Sex (1871) that I began to understand why the paralysis and the heart attacks might be worth the risk. Darwin’s theory of natural selection concerns the evolution of new species by selection of traits that adapt to the environment across generations. But his theory of sexual selection concerns the emergence of traits and behaviours that enhance the ability to find a mate and reproduce.

Darwin’s theory of sexual selection was, in fact, an answer to critics of his theory of evolution, who pointed out that not all traits helped individuals survive. Darwin explained the phenomenon by pointing to the peacock’s brightly coloured tail and the reindeer’s elaborate antlers, with their unwieldy and dangerous exfoliations. How could such risky accoutrements make these animals fit enough to outcompete predators in the struggle for existence? Their very adaptations, those glaringly identifiable tails and those massively unwieldy antlers, seemed to hinder them in escaping their predators – not to mention the energy requirements these traits impose.

REM paralysis is in an evolutionary protection, put in place to keep us from acting out our dreams and harming our partners

However, Darwin pointed out that many features of sexually reproducing species can boost reproduction rather than survival in the environment per se. The peacock’s tail advertised its fitness to peahens, and so they tended to mate with the male who had the most extravagant tail in the group. Any peacock whose genes could support such a costly tail must be fit indeed! Similarly, the reindeer’s antlers were used as weapons in the fight against other males of the same species for access to females. The more elaborate the antlers, the more forbidding the buck. The goal was to intimidate your competitors enough that they would give up the fight for access to a fertile female and you would then win the chance to mate with her. The pressures of sexual selection made it imperative that males develop weapons such as aggressiveness, antlers, body armour, muscles, and stingers to fight other males for access to females as well as elaborate adornments to attract females. Thus, the costly, apparently non-adaptive traits were explained.

If we accept the sexual selection theory of REM sleep and dreams, we might suppose that there are some underlying advantages reflected in adapting to the outside world. Take the REM-related lapse in the ability to generate body heat. The vast majority of animals, including humans, sleep either with relatives or with sexual partners. Co-sleeping and huddling together allows for preservation and generation of heat between partners, preventing metabolic waste – and also creating opportunities to reproduce.

A similar advantage can be found in REM paralysis. To get to the root of things there, researchers have studied REM Behaviour Disorder, in which the neurons that cause the paralysis have been destroyed, allowing dreamers to move about. Such patients, most of them male, dream that they or their wives are under attack from other men, and physically defend themselves while asleep with punches and kicks. REM paralysis, therefore, is in some sense an evolutionary protection, put in place to keep us from acting out our dreams and harming our partners.

Other researchers have asked what, if any, effect REM dreams have on our behaviour in waking life. Dylan Selterman, a social psychologist at the University of Maryland, has addressed these questions directly. With his team, he asked participants who were in committed relationships to keep track of all their dreams, as well as their daily activities, for two weeks. The researchers found that dreaming about some sort of interaction with a romantic target significantly predicted romantic interactions with others the next day. For example, if the dreamer dreamed of having sex with his or her partner, they reported more love and intimacy the next day – but only if the relationship was going well. Similarly, if the dreamer dreamed of conflict with a partner during the night, greater conflict was experienced in that relationship the next day.

On one level, these results appear to confirm the obvious – namely that dreams can influence our waking behaviours. However, what the theory of sexual selection adds to a commonsense interpretation is that dreams prefer to influence sexual behaviours as opposed to behaviours overall. That is what they were designed to do.

Of course, for absolute proof that dreams are a linchpin of reproductive fitness we must show enhanced fertility in those who dream more. The experiment would be massive, requiring we correlate variations in fertility rates across large numbers of dreamers from many different cultures to eliminate competing explanations. That study has begun, with help from dreamboard.com, an app that aims to help users record and understand their dreams and contains more than 200,000 dreams from people all over the world. Several thousand dreams are added to the site every couple of weeks.

But we don’t need to finish that project to know that the mind is not merely some information-processing device producing unbiased perceptions of the world. Instead, it is infused with and driven by sexual desire – just like my hormone-addled teenage self. Freud was the first to see the psyche as an arena of male-female conflict and co-operation propelled by long-term sexual strategies. My work on dreams is now confirming this central insight. Dreams are products of, and contributors to, the ancient war and dance between the sexes. That eternal dance is written into the flesh and blood of the brain and re-enacted nightly in the turbulent realm of the dream. What stuff, then, are dreams made on? Not just sexual desires aimed at romantic targets, but aggressive acts aimed at competitors as well. We not only dance in our dreams: we also become raging bulls.