The idea of summoning the spirits took thrilling hold of the Victorian imagination – and has its adherents now. But the psychology behind spiritualism is more intriguing

As the evenings get darker and the first hint of winter hangs in the air, the western world enters the season of the dead. It begins with Halloween, continues with All Saints' and All Souls' days, runs through Bonfire Night – the evening where the English burn effigies of historical terrorists – and ends with Remembrance Day. And through it all, Britain's mediums enjoy one of their busiest times of the year.

People who claim to contact the spirit world provoke extreme reactions. For some, mediums offer comfort and mystery in a dull world. For others they are fraudsters or unwitting fakes, exploiting the vulnerable and bereaved. But to a small group of psychologists, the rituals of the seance and the medium are opening up insights into the mind, shedding light on the power of suggestion and even questioning the nature of free will.

Humanity has been attempting to commune with the dead since ancient times. As far back as Leviticus, the Old Testament God actively forbade people to seek out mediums. Interest peaked in the 19th century, a time when religion and rationality were clashing like never before. In an era of unprecedented scientific discovery, some churchgoers began to seek evidence for their beliefs.

Salvation came from two American sisters, 11-year-old Kate and 14-year-old Margaret Fox. On 31 March 1848, the girls announced they were going to contact the spirit world. To the astonishment of their parents they got a reply. That night, the Fox sisters chatted to a ghost haunting their New York State home, using a code of one tap for yes, two gaps for no. Word spread and soon the girls were demonstrating their skills to 400 locals in the town hall.

Within months a new religion had emerged – spiritualism – a mixture of liberal, nonconformist values and fireside chats with dead people. Spiritualism attracted some of the great thinkers of the day – including biologist Alfred Russel Wallace and Sir Arthur Conan Doyle, who spent his latter years promoting spiritualism in between knocking out Sherlock Holmes stories. Even the admission of the Fox sisters in 1888 that they had faked it all failed to crush the movement. Today spiritualism thrives in more than 350 churches in Britain.

The tricks and techniques used by mediums have been exposed many times by people such as James Randi, Derren Brown and Jon Dennis, creator of the Bad Pyschics website.

Last week I spent 40 minutes with a telephone spiritualist who passed on messages from four dead people. Like all mediums, she was skilled at cold reading – the use of probable guesses and picking up of cues to steer her in the right direction. If she hit a dud – the suggestion that she was in the presence of a 40-year-old uncle of mine – she quickly widened it out. The 40-year-old became an older person who felt young at heart. And then someone who was more of an uncle figure. She was also skilled at the Barnum effect – the use of statements that tend to be true for everyone.

Among dozens of guesses and misses, there was just one hit – the correct name of a dead relative. Their relation to me was utterly wrong, as were details of their health. But the name was right and, even though it was a common name among that person's generation, it was a briefly chilling moment.

Professor Richard Wiseman, a psychologist and magician, says my response to this lucky guess is typical. People tend to remember the correct details in a seance but overlook statements or events that provide no evidence of paranormal powers.

TV illusionist Derren Brown has often used his act to denounce paranormal practices. Photograph: David Yeo

Wiseman's work has also shown that we are all extremely susceptible to the power of suggestion. With colleague Andy Nyman, co-creator of Derren Brown's television illusions, Wiseman used contemporary descriptions of Victorian seances to recreate an encounter with spirits in a disused prison. Over eight seances involving 152 people, volunteers sat around a table in the dark holding hands while luminous painted bells, balls and maracas moved before their eyes. Surveyed afterwards, a fifth of the volunteers believed they had witnessed the paranormal.

"These things are often very simple," says Wiseman, author of Paranormality. "We had a man creeping around with a stick. We thought when we read the original accounts of how seances were carried out that they wouldn't fool anyone. We were wrong. A lot this is do with framing. Once you think you have an explanation for an event you don't have any other ones. Once you think it's a spirit you don't look for another explanation."

During the seance, Nyman, taking the role of the medium, announced that the spirit would raise the table. Soon afterwards he encouraged the spirit by saying "lift the table higher" and "the table is moving now". Two weeks later a third of the participants recalled wrongly that the table had moved.

"Suggestion builds over time. If you ask people immediately after the event it is not so effective. You don't want to solidify the memory immediately after the event," says Wiseman.

The trappings of the seance increase its success. Holding hands prevents participants from disrupting the trickery. Darkness increases sensitivity to sound and movement and makes people more scared – which may, Wiseman says, increase susceptibility.

The seance can be explained by stage magic and human frailty. But what about phenomena such as table tipping and Ouija boards?

Table tipping, or turning, has gone out of fashion but is easy to replicate with four or more people, a small table, dim lights and a relaxed atmosphere. The group place hands on the table and wait. After 40 minutes or so the table should start to move. It soon appears to have a mind of its own, sliding, swaying and even pinning people to the walls.

The reason why household furniture can appear to be possessed was exposed more than 160 years ago by Michael Faraday, the discoverer of the link between magnetism and electricity. In 1852 Faraday was fascinated by the new craze of table tipping – and whether people or spirits were responsible. So he took bundles of cardboard roughly the size of a table top and glued them weakly together. Each sheet got progressively smaller from top to bottom, allowing Faraday to mark their original positions on the card above with a pencil. He then placed the cards on a table and asked volunteers to put their hands on the cards and let the spirits move the table to the left.

Ouija boards were debunked by psychologist Joseph Jastrow in the 1890s. Photograph: Corbis

This experiment allowed Faraday to see what was moving the table. If it was spirits, the table top would slide out the cards from the bottom up. But if the participants were doing it, the top cards would be the first to move. By examining the position of the pencil marks Faraday showed that people, not spirits, moved the table. He had demonstrated the ideomotor response, the movement of muscles independent of deliberate thought. This also explains table tipping's sophisticated big brother, the Ouija board.

In a Ouija seance participants place fingers on a glass on a table surrounded by letters and watch as it eerily moves – and occasionally spells out words. Psychologist Susan Blackmore is best known as the proponent of memes, but early in her career she was a parapsychologist. At Oxford she ran the student Psychical Research Society, carrying out experiments using Ouija boards. Time and again the glass spelled words and sentences. Her confidence began to be shaken when she modified the board.

"We turned the letters upside down because surely spirits should see the letters underneath," says Blackmore, now a sceptic. "And of course it spelt out rubbish. It cannot work unless all the people can see what is going on."

The ideomotor effect is also at play with the glass. "With a Oujia board, your arm is getting tired and your ability to judge the location of your finger is compromised," says Blackmore. "When the glass moves you naturally adjust your movements and go along with the glass. To start with it moves hesitantly, but after a while as soon as it starts moving everyone's hand follows.'

But what about the glass's ability to spell? That was investigated by the American psychologist Joseph Jastrow in the 1890s. He used a device called the automatograph made of two glass plates separated by brass balls. Any involuntary movement of hands placed on the top plate causes it to move. The movement is recorded by a pencil attached to the device.

When Jastrow asked volunteers to imagine looking at an object in the room the automatograph revealed that their hands involuntarily moved in that direction. Just visualising the door was enough for the hands to drift towards it.

And that's what's happening with a Ouija board. If the participants look at a particular letter – because they expect it to follow next – they unwittingly nudge the glass towards it.

If the Ouija board has shed light on unwitting movement, then another technique, channelling of spirits, has questioned free will.

Harvard psychologist Dan Wegner, who died this year, is best known for his work on the rebound effect. Tell someone not to think about white bears and they immediately think about white bears. The more we try to actively suppress a thought, the less likely we are to succeed. But he also investigated automatic writing, where people claim to write without being aware what they are doing.

The most famous automatic writer was Pearl Curran, an American who knocked out more than 5,000 poems, novels and plays while claiming to be channelling the spirit of Patience Worth, a 17th-century Englishwoman.

Automatic writing has traditionally been explained as the action of the subconscious mind. But Wegner argued that the reason lay in the illusion of free will. Most people have a sense of their inner you – the conscious self that makes decisions about day-to-day life. According to Wegner this sense is an illusion. There's evidence to back up this seemingly unlikely idea.

Sir Arthur Conan Doyle, who spent his latter years promoting spiritualism. Photograph: Getty Images/BBC

In the 1960s, neurophysiologist William Grey Walter got volunteers to operate a slide projector while their brain was monitored with electrodes. The participants were told to press a button to change slides. But the button was a fake – the projector was controlled by electrical activity in the brain. The startled volunteers found that the slide machine was predicting their decisions. A fraction of a second before they decided to press the button, the part of the brain responsible for hand movement burst into activity and – through the electrodes – moved the slide on.

Grey Walter showed that there was a fraction of a second delay between the brain making a decision and someone being aware that they were making a decision.

In the 1980s, Benjamin Libert of the University of California , San Francisco, made a similar discovery after attaching volunteers to electrical monitors and sitting them in front of a screen displaying a dot in a circle. The participants were told to flex their wrists whenever they liked, and report the position of the dots at the moment they made the decision to flex. Again, there was a surge in brain activity a fraction of a second before the volunteers were aware they were making a decision.

Wegner's solution was that our deliberate, thinking brain – the inner me that makes decisions – is an illusion. Instead, the brain does two things when it makes a decision to raise an arm. First it passes a message to the part in charge of creating the conscious inner you. Second, it delays the signal going to the arm by a fraction of a second. This delay generates the illusion that the conscious mind has made a decision.

Wegner argued that automatic writing occurs when something goes wrong with this process. The brain sends the signal to the arm to write – but fails to alert the inner you.

There's something a little ironic about his conclusion. The early spiritualists believed they were shedding light on the transition of the human spirit from the physical body to the afterlife. Wegner suggests that it's not just the distinction between mind and body that is false, but the whole concept of the "conscious" decision-making mind is just another piece of trickery played by the brain.

And meanwhile, 150 years after Faraday showed that table tipping was hokum, we continue to frighten one another in the dark.

'What is remarkable is that the stuff written in books 100 years ago still works,' says Richard Wiseman. 'If you think of all the technology and science and education and yet a group of people sitting in the dark can scare the living daylights out of themselves.'