"The light had been switched off, and a comfortable feeling of warmth and heaviness marked the onset of sleep. But suddenly a slurping sound startled the sleeper. Everything seemed completely normal – until he noticed the green liana which had grown from the pot plant and reached his body.

"Every movement became impossible, and every scream was smothered by the plant material that was growing around his mouth and throat. Thoughts raced through the mind: This wasn't a nightmare, the reality of the room was far too distinct and his perception and thinking were far too clear. Were the plants taking revenge on humankind, or had aliens conquered the planet?"

The above account sounds like a scene from a bad horror movie. But it isn't. Dr Stephan Matthiesen, a physicist at the School of Geosciences, University of Edinburgh, sent me the account last year saying he had personally experienced this terrifying scenario and lived to tell the tale. It's taken from his book The Normality of Altered States of Consciousness.

Dr Matthiesen is a scientist and looked for a rational explanation of what had happened. He was right do so, because what he had experienced was an example of a surprisingly common phenomenon known as sleep paralysis. Many similar experiences have been reported to the Anomalistic Psychology Research Unit I coordinate at Goldsmiths, University of London.

One of our students, Peter Moore, used to suffer from sleep paralysis on a regular basis. One night, for example, he awoke to find himself unable to move and with a strong feeling of tension across his chest, making it almost impossible to breathe. He could see his bedroom and managed to tilt his head, only to see an evil-looking black cat sitting there hissing at him.

But what was most terrifying about this vision was that the cat's white skull was inverted and dripping some sort of black goo. By a huge effort of will, he finally managed to break out of his paralysed state with the intention of attacking his visitor, only to find himself delivering a right hook to thin air.

Jeremy Deane, a fellow student at Goldsmiths still suffers regular attacks of sleep paralysis, particularly if his usual sleep pattern has been disrupted. As is often reported, he can experience multiple episodes in a single night. His experiences typically involve paralysis, difficulty breathing, strange proprioceptive hallucinations such as his body vibrating, and bizarre "hyper-real" visual hallucinations during which objects may metamorphose into nightmarish objects.

For example, clothes lying on the floor may become dead bodies or a ceiling fan might turn into a "faery" with the blades as wings and the central bulb as an animate, speaking face. In his own words: "The experience is usually terrifying, but I have been able to control it and sometimes it has been very pleasant (occasionally there can be a sexual element, or 'floating' feelings accompanying what appears to be an out-of-body experience).

"Common images are bearded, goblin-like demons laughing or whispering sinister speech, a faceless girl (usually covering her face with hair, moving around in bed moaning and feeling my body), hands appearing from the wall and attempting to strangle me. A hung man talking in the corner of the room, and some of the most bizarre experiences may include up to a dozen 'critter' entities (think Gremlins movie) laughing and talking about me. The environment tends to feel like a holographic dollhouse, the experience peaks and then the hallucinations mysteriously vanish when I regain control of my body."

There are numerous descriptions of sleep paralysis in works of fiction, from Herman Melville's Moby Dick to accounts by F. Scott Fitzgerald and Ernest Hemingway. There are also the classic artistic depictions painted by Henry Fuseli in the late 18th century.

Not surprisingly, episodes of sleep paralysis are often interpreted by the sufferer in paranormal terms, but sleep researchers have provided a more convincing account.

During normal sleep, the brain and body pass through four stages of sleep during which physiological indices such as brain activity, heart rate and breathing rate gradually slow down. This process then reverses and the sleeper enters a period of REM-stage sleep, so-called because of the characteristic rapid eye movements that are associated with it.

The full cycle, which takes about 90-100 minutes, is then repeated and, as the night progresses, each cycle becomes progressively less dominated by stage 3 and stage 4 sleep and increasingly dominated by REM sleep. Dreams can occur during all sleep stages but the most vivid dreams tend to be reported when people are awoken from REM sleep.

Throughout these REM periods the muscles of the body are paralysed, presumably to prevent the dreamer from acting out the dream. During sleep paralysis episodes, however, something goes awry with the normal process and the individual becomes aware of the fact that they cannot move. This can be scary enough, but in a minority of cases, such as those described above, this curious mix of normal wakeful consciousness and dream consciousness can result in bizarre and often terrifying hallucinations.

Our own research confirms the results of previous surveys in showing that sleep paralysis in its most basic form is surprisingly common. Although sleep paralysis is a common symptom of narcolepsy (a condition characterised by uncontrollable bouts of deep sleep), around 40% of the rest of the population report that they have had the experience.

Around one in 20 people report experiencing associated symptoms, including a strong sense of a presence, difficulty breathing due to pressure on the chest, intense fear, and a wide range of hallucinations. These can be visual (lights or dark shadows moving around the room, monstrous figures), auditory (voices, footsteps, mechanical sounds), tactile (feeling that one is being touched or dragged off the bed or that the bedclothes are being pulled off), and/or proprioceptive (body or limbs vibrating, rapid acceleration of the whole body, out-of-body experiences).

Fortunately, most people never experience sleep paralysis, and of those that do, most only experience it once or twice in their lives, usually in its most basic form. Furthermore, most people in modern Western societies, although somewhat shaken by the experience, just shrug it off as some kind of nightmare. But a small minority assume that either the events they experienced were real, or else they are losing their sanity.

If you are one of those people, possibly suffering in silence because of fear of being ridiculed or even being treated for a psychiatric disorder, please be reassured. You are not the victim of nocturnal attack by spirits or attempted abduction by aliens. Neither are you at greater risk than the general population of serious psychopathology.

Although most people do not opt for a paranormal interpretation, the experience is so common it only requires a small percentage of sufferers to do so to account for the very large number of claimed paranormal encounters.

Sufferers can be just as terrified by the experience even if the episode does not involve any ostensibly paranormal content, sometimes experiencing their unwanted intruder as a burglar, a murderer or a rapist. Even sufferers who are well-informed about sleep paralysis and do not experience the more florid symptoms described above still experience intense fear unlike anything they experience in waking life.

This strongly suggests that the fear is not a consequence of the experience but an integral part of it, possibly caused by over-activation of the amygdala, the part of the brain that is responsible for fear.

This is illustrated by an account from Lori Ball, a healthy 53-year-old woman from Ohio who is not only well-informed about sleep paralysis but is actually cognisant of what is happening to her while it's happening: "I try to scream (though I have great difficulty making any sound), attempt to flail around, anything, to get the attention of my husband. It is a feeling of panic, entrapment and desperation so horrifying that I have difficulty describing its magnitude.

"If my spouse notices my discomfort and responds, in my mind it's never soon enough. One cannot simply tell me to 'wake up' and tap me on the arm. Often I need to be shaken somewhat to be fully present. At that point I wouldn't care if he slapped me hard as the terror of being in that paralysed state, totally helpless, is overwhelming. Knowing that it will end eventually is of no comfort. Every second is hell."

One of the most fascinating aspects of sleep paralysis for me is the different ways that the same core experience is interpreted across different cultures. Accounts from Europe in the Middle Ages indicate that sleep paralysis episodes were often interpreted in terms of nocturnal visits by witches or sex-crazed demons.

Even today, many societies interpret such experiences in supernatural terms based upon folklore. In Newfoundland, belief in the "Old Hag" who sits on the sleeper's chest and suffocates them is common. The Japanese speak of kanashibari, a type of nocturnal spiritual attack.

In St Lucia, the same core experience is explained as the souls of unbaptised children who crawl on to the sleeper's chest and throttle them.

Over the summer, I visited Taiwan to set up a cross-cultural study of sleep paralysis with colleagues at Kaohsiung Medical University. In China and Taiwan, sleep paralysis attacks are often referred to as "ghost oppression" and levels of supernatural belief are very high among the Taiwanese population.

In our study, we are interested not only in the degree to which such high levels of belief affect the tendency to interpret the experience in supernatural terms but also the degree to which pre-existing belief systems can affect the content of the hallucinations themselves. Sleep paralysis offers an almost unique opportunity to study the reciprocal interaction between biology and culture.

It seems likely that the core experience has itself played a role in the development of belief systems relating to the spirit world in many cultures and that those very belief systems, once elaborated upon, are then capable of influencing the hallucinatory content of sleep paralysis episodes in subsequent generations.

Chris French is a professor of psychology at Goldsmiths where he heads the Anomalistic Psychology Research Unit. He talks about sleep paralysis on The One Show tonight on BBC1 at 7pm