Here I am, lying in bed. If you walk in now, you’ll think I’m sleeping. But I see you. Although my eyelids look shut, they are fluttering slightly. They are the only parts of me that I can move. I am fully conscious but I cannot shout out to you: my body is completely frozen.

Everybody is paralysed during rapid eye movement (REM) sleep, the stage of sleep where dreaming occurs. If we weren’t paralysed, we would act out our dreams, endangering ourselves and our sleeping partners. But sometimes, especially when sleep patterns are disrupted or we get exhausted, things go awry: REM extends into waking consciousness, our bodies become immobile and our alert brains fuse with the imagery of dreams. The phenomenon of waking up during REM, completely unable to move, is called sleep paralysis.

The experience can be terrifying. Trapped in your paralysed body, you might sense the presence of a malevolent intruder in the room or a pressure on your chest, squeezing the breath out of your lungs. Hallucinations can jangle the senses: there are ominous voices, supernatural entities, strange lights. You feel as if you are being touched or dragged, bed covers seem to be snatched from you, and you are helpless to grab them back.

I have experienced the frightening imagery of sleep paralysis since childhood, but only later did I understand that my dark journey was not unique – I share it with at least 6 per cent of people worldwide, and it has been reported for thousands of years as encounters with sexual demons, beasts, and ghosts. These reports differ by culture – but the texture and the biology is the same. From Newfoundland come tales of the Old Hag, a hideous witch who pins down sleepers by sitting on their chests. Japanese folklore gives us kanashibari, the fate of the unfortunate or cursed who have been magically tied up in their sleep by evil spirits. In Old Norse, the Mara is a malevolent spirit who straddles the body of the sleeper as if riding a horse, then tries to strangle them; mara is the origin of the English word ‘nightmare’. UFO abduction stories and alien encounters likely emerge from sleep paralysis, too.

Ever since I was a teen, I have seen shadow figures in the corner of my bedroom, and awoken to find strange entities – grinning vampires or silent watchers – by my bed. I’ve felt my hand grasped, my chest crushed by the weight of a strange beast; my body twisting and spinning in space. I’ve heard buzzing, ringing, whooshing and nasty names whispered in my ear. If the radio or TV were on, I could hear the programmes clearly and, after paralysis released me, I could report them back. If someone walked into the room, or the doorbell rang, or a dog barked, or (as happened on one occasion) there was a power outage, I was fully aware. I tried to shout out, to pull at my eyelids, desperate to snap out of it, but I could not budge.

With this ghoulish treasure trove to draw upon, sleep paralysis has naturally spawned some very scary stories and films. But as a writer and filmmaker as well as a long-time percipient, I have another story to tell. Beyond the sheer terror, sleep paralysis can open a doorway to thrilling, extraordinary, and quite enjoyable altered states. One is the lucid dream state, in which you can consciously manipulate your dreams, traversing incredible landscapes and interacting with creatures conjured in your mind. Another is the out-of-body experience – the waking sensation of separating from your physical body and floating, spinning and flying through your surroundings; often, you’d look down to see yourself lying below.

The biological underpinnings of sleep paralysis have become less mysterious in recent years. The psychologist Kazuhiko Fukuda at Edogawa University in Japan explains the likely involvement of the amygdala, a brain region that signals fear from threats in the environment and triggers our primal ‘fight or flight’ reactions. Waking up paralysed constitutes an environmental threat, yet we cannot react. The amygdala is in hyperdrive, and REM physiology has invaded our consciousness. We are left stuck in a state of overwhelming terror, leaving us dreaming awake and set upon by our deepest fears.

In 2012, University of Toronto neuroscientists Patricia Brooks and John Peever reported the physiological process behind the altered state: GABA A and GABA B , the receptors that regulate the body’s muscle tone, combine with glycine, an amino acid, to switch off motor-neurone activity in our voluntary muscles during REM sleep. Normally, they switch our motor-neurone activity back on before we wake up. But, sometimes, we wake up during REM, and the GABA and glycine keep hold of us – the scary result is dreaming awake.

I could float up to my bedroom ceiling or into the living room or out through the solid front door

One of the most probing explorations of this state, and the one that helped free me from the terror, comes from Jorge Conesa-Sevilla, a neurocognitive psychologist and shamanic artist based in Oregon who regularly experiences sleep paralysis himself. In his book Wrestling with Ghosts (2004), he takes a refreshing approach to the subject, couching sleep paralysis in scientific terms, without denying his personal, exploratory approach.

Conesa-Sevilla taught me that people who experience sleep paralysis have a unique advantage in dreaming lucidly – they can use their altered state as a launch pad for full-blown dream control. It makes sense: both lucid dreams and sleep paralysis are ‘blended states’, according to the psychologist James Cheyne of the University of Waterloo in Canada – but these states are distinct. ‘Lucid dreaming seems to consist of waking awareness intruding into dreams and sleep paralysis of dream imagery intruding into waking consciousness.’

Conesa-Sevilla has developed specific, highly honed techniques to help us move from one blended state to the other. Like many others who regularly experience sleep paralysis, I had naturally slipped into lucid dreams on occasion, but I did not understand what they were, or that I could initiate this switch. Wrestling with Ghosts explained how to do this, but most importantly, it made me understand that sleep paralysis was not a curse; it could be a gift.

Conesa-Sevilla’s system, called Sleep Paralysis Signalling (SPS), is used to acknowledge and exploit your self-awareness in order to transition from one altered state to the other: from terror to bliss. It includes focusing on particular parts of your body, imagining that you are spinning, and using meditation, controlled breathing and relaxation for managing the fear of the paralysed state. Tapping SPS, I can wilfully go from waking to the dream state, retaining just enough consciousness to influence the action within.

To switch from sleep paralysis into lucid dreaming is no mean feat; it is hard to keep a cool head with a ghost sitting on top of you. I can rarely pinpoint the moment that terror becomes lucidity but, when it does, I am launched into the vast landscapes and vivid colours of my lucid dreams.

I often return to the same places, worlds that I have created. There is a city with a complex network of streets, elaborate houses, an underground system, a harbour and swimming pools. The whites, blues, yellows and greens are far more intense than any I have seen in waking life. And there are great natural landscapes: a coastline with high cliffs and forests. I know my way around. I could draw a map of these worlds. I can choose where to go and I can walk or fly. I populate these landscapes with people; be they familiar or fantastical, living or departed, I talk to them. I am fully conscious during these dreams.

My lucid dreams are often accompanied by sensations of flying, floating or leaping across the landscape. But sometimes I have another experience, similar in that it is characterised by flying and floating sensations, yet distinct. During a lucid dream I am ‘intact’ and moving around a dreamscape, whereas during these other experiences, I seem to physically twist or stand up and ‘out of myself’ and into my immediate surroundings. This sensation feels as real to me as it would if I were to stand up now – and it is experienced as fully alert consciousness. I now understand this to be a form of out-of-body experience, or OBE.

With hindsight, I realise I’ve been having OBEs for some time. In one childhood memory, I’m lying both ‘on’ and ‘under’ my bed at the same time. Later, I willed the experience out of terror during the sleep paralysis itself. If I yell, but make no sound, I thought, if I feel, but nothing is touching me, if I move my arms, but they are still, then my paralysed body is, somehow, receiving sensations of movement from my brain. What would happen if I consciously willed this phenomenal body to twist out of my paralysed body? And I found that, in my mind at least, I could.

At first there were loud noises, buzzing and whooshing. At times it felt as if my brain was being sucked out of the top of my head, or that my whole body was being pulled backwards at high speed. I would panic and fight it, but each time I became a little braver. I would ride out the scary sounds and sensations, and find that they gave way to a pleasant feeling of being completely separate from my body.

I could see my bedroom, but in altered form. The plain wooden door had beautiful paintings on it; the trees in the garden were a different species or larger than normal. At times I seemed to be dragging myself around; at others I was light and moved with ease.

During these OBEs, I wondered what would happen if I tried to push my body through my closed bedroom door, and I found that I could. I started to play with these sensations, to float up to my bedroom ceiling or into the living room or out through the solid front door. I enjoyed the feeling of spinning around my house and garden.

My lucid dreams and OBEs are delightful because I can consciously heighten my experience, and a little terror can be exhilarating

I understand the mind and body to be a complex biological and chemical entity, intertwined, yet my hallucinogenic nights suggested otherwise. What was this ‘self’ that appeared to get free? I was unnerved by a friend who suggested I stop leaving my body lest a ‘lost soul’ inhabit it while I was gone, blocking my return. But my fears were eased by talks with the experts. The neuroscience fascinated me, and set me free.

Our sense of ‘self’ as embodied, moving around space relative to gravity, comes from sensual input: spatial position and balance, touch and movement, and visual cues. These ‘vestibular’ sensations, coming from diverse neural networks in different parts of the brain, are brought together and processed at the junction of the temporal and parietal lobes (or our temporal-parietal junction), a region just above and behind the ears. When we are awake, our temporal-parietal junction is highly active, and it processes information efficiently and coherently. During dream sleep, however, vestibular sensations come not from the external environment but from within the brain itself.

During REM, vestibular sensations might be the source of those lovely flying dreams. But they can also be felt during sleep paralysis – and can be used to propel us to OBE. The jet fuel here is dissonance: from REM, we get vestibular sensations produced by the brain and from waking we get vestibular input from the outside world; both land in the temporal parietal junction. During sleep paralysis, Cheyne explains, the brain tries to reconcile the conflict ‘between movement and non-movement, between simultaneously floating above, and lying on, one’s bed’. He thinks we resolve the conflict by ‘a splitting of the phenomenal self and the physical body, sometimes referred to as an out-of-body experience’.

The notion is supported by Olaf Blanke, professor of neurology at the University Hospital of Geneva, whose studies over the past decade have helped to transform the field. Blanke has shown when he stimulates the brain with electrical current, it generates its own vestibular sensation and transports it to the temporal-parietal junction, recreating the pattern that occurs during REM. As a result, the sense of self as embodied is lost and the individual reports feeling separate and detached. Of special note is the position of the temporal-parietal junction just above and behind the ears, perhaps an explanation for the sense of a presence that seems to lurk behind or just out of sight – the very entities so many of us describe as part of the sleep-paralysis experience.

Some might think that these neurological explanations for sleep paralysis, lucid dreams and OBEs are impediments to the experience but, to me, they enrich it. My lucid dreams and OBEs are even more delightful because I can consciously heighten my experience, and a little terror can be exhilarating. To this day, I’m frightened when I wake up paralysed. After all, my amygdala is screaming FEAR! FEAR! FEAR! But, with my new-found understanding, I can overcome the terror and take advantage of being awake to explore these altered states. The transition from state to state can be slippery but, the more I understand what is happening in my brain, the more control I have and the more enjoyable the experience becomes.

Here I am again, lying in bed. If you walk in now, you’ll think I’m sleeping. But I am not: I am conscious and I am flying, bounding across landscapes coloured by dreams.

If you ever wake up unable to move, try not to panic. Remind yourself that you stand at the threshold of a fantastical world, a strange hinterland, an exhilarating space in which you are awake, but have a REM toy box at your disposal.