How do we deal with a purposeless universe and the finality of death? From Victorian séances to the embalming of Lenin's corpse to schemes for uploading our minds into cyberspace, there have have been numerous attempts to deny man's mortality. Why can't we accept the limits of science?

The séance that Charles Darwin attended in January 1874 at the house of his brother Erasmus brought the pioneering biologist together with Francis Galton, eugenicist and one of the founders of modern psychology, and the novelist George Eliot. All three were anxious that the rise of spiritualism would block the advance of scientific materialism. They were unimpressed with what they witnessed – Darwin found the experience "hot and tiring" and left before sparks were seen and rapping heard – but they would have been seriously concerned had they known the future career of a fourth participant in the séance, the classical scholar and psychologist FWH Myers.

The inventor of the word "telepathy" and the writer who first introduced the work of Freud into Britain, Frederic Myers went on to become one of the founders of the Society for Psychical Research. Supported by some of the leading figures of the day, including the Cambridge philosopher Henry Sidgwick and Arthur Balfour, president of the society and later prime minister, the psychical researchers believed human immortality might prove to be a scientifically demonstrable fact.

Their quest for an afterlife was partly driven by revulsion against materialism. Science had revealed a world in which humans were no different from other animals in facing oblivion when they died and eventual extinction as a species. For nearly everyone the vision was intolerable. Not fully accepted by Darwin himself, it led the biologist and explorer Alfred Russel Wallace – acknowledged by Darwin as the co-discoverer of natural selection – to become a convert to spiritualism. Wallace insisted he did not reject scientific method. Like Sidgwick and Myers, he was convinced science could show the materialist view of the world to be mistaken.

Very often these Victorian seekers also had other, more personal motives. Members of an elite that protected itself from scrutiny by keeping to a code of secrecy, leading psychical researchers used their investigations to reveal, and then again conceal, aspects of their lives that they or their culture could not or would not accept. For Myers the search for evidence of survival became a passion when a married woman for whom he had formed a deep attachment committed suicide, leading him to spend the remainder of his life trying to contact her through mediums. Sidgwick spent decades earnestly searching for proof of life after death because without this evidence, he believed, there is no reason for living a moral life. If the visible world is the only reality, he wrote at the end of Methods of Ethics (1874), morality is "reduced to chaos". He excised this passage from subsequent editions of the book, but never altered his view. Sidgwick feared the finality of death because it left no reason to restrain one's desires – a thought he must have found extremely troubling, since he seems to have spent much of his life suppressing a part of his sexuality.

Balfour was celebrated for his aloof detachment. Yet through his brother Gerald, a former Conservative minister who gave up politics to study the paranormal, the former prime minister entered into what purported to be a correspondence with a long-dead woman, whom many believed he had once loved. Balfour's ostensible correspondent communicated by automatic writing – texts produced without conscious awareness in which another mind seems to be guiding the pen, which became a vehicle for unresolved personal loss and secret love.

Starting early in the 20th century, tens of thousands of scripts were produced by different mediums in several countries over a period of more than 30 years. Known as the "cross-correspondences" because they seemed to be linked together, the scripts contained texts claiming to be messages from deceased psychical researchers, including Sidgwick and Myers, which together demonstrated the reality of life after death. As the flow of scripts continued an even larger claim emerged: the dead had taken on the task of saving the world of the living by means of a post-mortem experiment in eugenics. Scientists who had passed to "the other side" were fashioning an exceptional human being, a posthumously designed messiah-child who would deliver humankind from chaos and bring peace to the world.

A child was in fact born – the offspring of Balfour's brother and the medium who transcribed the scripts, the wife of a much older man who took up automatic writing under the cover of a pseudonym after her daughter died in infancy – but seems to have known nothing of the role he had been assigned until late in life, and then probably less than the whole truth. Featuring a spell in MI6 (where for a time he worked alongside Kim Philby) followed by life in a monastery, the career of the supposed messiah was certainly unusual. But he had no impact on the world at large, which continued its normal course of conflict and drift.

The idea of dead scientists engaging in an experiment in eugenics is incredible enough. Yet the most striking feature in this episode – only fully revealed more than 100 years after the scripts began to appear – is the power that is ascribed to science itself. While spiritualism evolved into a popular religion, complete with a heavenly "Summerland" where the dead lived free from care and sorrow, the intellectual elite of psychical researchers thought of their quest as a rigorously scientific inquiry. But if these Victorian seekers turned to science, it was to look for an exit from the world that science had revealed. Darwinism had disclosed a purposeless universe without human meaning; but purpose and meaning could be restored, if only science could show that the human mind carried on evolving after the death of the body. All of these seekers had abandoned any belief in traditional religion. Still, the human need for a meaning in life that religion once satisfied could not be denied, and fuelled the faith that scientific investigation would show that the human story continues after death. In effect, science was used against science, and became a channel for belief in magic.

Much of what the psychical researchers viewed as science we would now call pseudo science. But the boundaries of scientific knowledge are smudged and shifting, and seem clear only in hindsight. There is no pristine science untouched by the vagaries of faith. The psychical researchers used science not only to deal with private anguish but also to bolster their weakening belief in progress. Especially after the catastrophe of the first world war, the gradual improvement that most people expected would continue indefinitely appeared to be faltering. What had been achieved in the past seemed to be falling away. If the scripts were to be believed, however, there was no cause for anxiety or despair. The world might be sliding into anarchy, but progress continued on the other side.

Many of the psychical researchers believed they were doing no more than show that evolution continues in a post-mortem world. Like many others, then and now, they confused two wholly different things. Progress assumes some goal or direction. But evolution has neither of these attributes, and if natural selection continued in another world it would feature the same random death and wasted lives we find here below.

Darwinism is impossible to reconcile with the notion that humans have any special exemption from mortality. In Darwin's scheme of things species are not fixed or everlasting; there is no impassable barrier between human minds and those of other animals. How then could only humans go on to a life beyond the grave? If all life were extinguished on Earth, possibly as a result of climate change caused by humans, would they look down from the after-world, alone, on the wasteland they had left beneath? Surely, in terms of the prospect of immortality, all sentient beings stand or fall together. Then again, how could anyone imagine all the legions of the dead – not only the human generations that have come and gone but the countless animal species that are now extinct – living on in the ether, forever?

Science could not give these seekers what they were looking for. Yet at the same time that sections of the English elite were looking for a scientific version of immortality, a similar quest was under way in Russia among the "God-builders" – a section of the Bolshevik intelligentsia that believed science could someday, perhaps quite soon, be used to defeat death. The God-builders included Maxim Gorky, Anatoly Lunacharsky, a former Theosophist who was appointed commissar of enlightenment in the new Soviet regime, and the trade minister Leonid Krasin, an engineer and disciple of the Russian mystic Nikolai Fedorov, who believed that the dead could be technologically resurrected. Krasin was a key figure in the decisions that were made about how Lenin's remains would be preserved.

Weakened in Britain, belief in gradual progress had ceased to exist in Russia. An entire civilisation had collapsed, and the incremental improvement cherished by liberals was simply not possible. The idea of progress was not abandoned, however. Instead it was radicalised, as Russia's new rulers were confirmed in their conviction that humanity advances through a succession of catastrophes. Not only society but human nature had to be destroyed, and only then rebuilt. Humans did not go on to a new life on the other side. There was no other side. When humans died they returned to dust, just like other animals. But once the power of science was fully harnessed, the God-builders believed, death could be overcome by force. Eventually all of humankind could look forward to scientifically guaranteed immortality, but the process of technological resurrection would begin with the most valuable of human beings – Lenin.

The poet Mayakovsky captured the mood among Bolsheviks when Lenin's death was announced on 21 January 1924: "Lenin, even now," he wrote, "is more alive than all the living." For Krasin this was more than a poetic conceit. Soon after Lenin's funeral he published an article in the communist newspaper Izvestia entitled "The Architectural Immortalisation of Lenin". After deliberations involving Stalin and the head of the secret police, Felix Dzerzhinsky, who had organised the funeral, it had been decided to embalm Lenin rather than bury or cremate the body. Krasin wanted Lenin's mausoleum to be a site that surpassed Jerusalem and Mecca in grandeur and significance. In late March the funeral commission that had been set up to organise Lenin's interment was renamed the immortalisation commission.

Lenin's tomb was designed by AV Shchusev, an architect involved in the constructivist movement and influenced by Kazimir Malevich, the founder of suprematism. Malevich viewed abstract geometrical forms as the embodiment of a higher reality. Believing that Lenin's cube-shaped mausoleum represented a "fourth dimension" where death did not exist, he suggested that Lenin's followers keep a cube in their homes. The proposal was adopted by the party, and cubic shrines to the dead leader were set up in "Lenin corners" in offices and factories. Shchusev's design reflected Malevich's belief in the occult properties of the cube. At a meeting of the funeral commission in January 1924, Shchusev declared: "Vladimir Ilyich is eternal . . . In architecture the cube is eternal. Let the mausoleum derive from a cube." He then sketched a design made of three cubes, which the commission accepted.

Krasin was also active. In a speech delivered at the funeral of a fellow revolutionary some years before Lenin's death, he had made clear his belief that in future, revolutionary leaders would not die forever: "I am certain that the time will come when science will become all-powerful, that it will be possible to recreate a diseased organism and resurrect great historical figures. I am certain that when that time will come, among the great figures will be our comrade." With this prospect in mind, towards the end of January 1924 Krasin constructed a refrigeration system designed to keep Lenin's cadaver cool. Unsurprisingly, the primitive cryogenic technology failed to work. The skin of the face had darkened, wrinkles were appearing and the lips had parted. Krasin was adamant that freezing could succeed if a better refrigerator was imported from Germany and double-glazing installed. But the process of deterioration continued, the nose began to lose its shape, one hand was turning greenish-grey, the eyes were sinking in their sockets and the ears were becoming crumpled.

Krasin's early experiment in cryogenics could not have succeeded. The doll-like facsimile that was pieced together from Lenin's earthly remains could never have been revivified. Even now, when cryogenic techniques are much advanced, the process of freezing is highly damaging to the cadaver. Krasin wanted to believe that the advance of knowledge made it possible for humanity to conquer death, but all science could do was fashion a lifeless dummy. Yet the god-builders did not renounce their faith that science would someday defeat death. When Krasin and Lunacharsky announced a competition for designs for a permanent shrine to replace the original wooden structure, they specified that the new mausoleum must include an underground chamber where the apparatus required for preserving Lenin's body would be housed.

Repeatedly re-embalmed, Lenin's enshrined corpse outlasted the Soviet regime. Extreme precautions were taken to secure its safety. When Nazi forces were approaching Moscow in 1941 the body was evacuated ahead of the city's living inhabitants. In 1973, when the Politburo decided to renew party documents, the first party card to be reissued was Lenin's. Throughout the last years of communism his suit was changed every 18 months. The process of rejuvenation continued after the communist collapse, and in 2004 it was announced that Lenin looked younger than he had done in decades.

There was logic in Lenin's immortalisation. He reacted furiously against any suggestion that Bolshevism was a new religion, writing to Gorky in 1913 that trying to construct a new god was an exercise in necrophilia. It was a shrewd observation, but Lenin was not as far from the god-builders as he liked to think. He too aimed to use the power of science to achieve the impossible – a materialist version of the earthly paradise promised in early Christianity. The Soviet experiment would bring into being not only a new society but a new kind of human being. It was a vision shared by HG Wells, who travelled to the Soviet Union to meet the Bolshevik leader. For Wells the new Soviet state was more than a political experiment. Having listened as a young man to the lectures of Darwin's fiercest disciple, TH Huxley, Wells was convinced that humankind would drift to extinction unless a conscious minority seized control of evolution. The Bolsheviks seemed to be doing exactly that, and when Wells met Lenin in 1920 he found the Soviet leader "very refreshing" – "a good type of scientific man". If the new Soviet state killed large numbers of people, Wells wrote, "it did on the whole kill for a reason and for an end". One of the intelligent few, Lenin was using his dictatorial power to fashion a new humanity.

While in Russia visiting Lenin, Wells stayed in Gorky's apartment, where he met the Russian writer's partner, a woman everyone called Moura, then 30 years old, previously the wife of a Baltic landowner killed in the revolution and for a time the lover of one of Britain's unofficial representatives in Russia. As Wells recalled in a suppressed section of his autobiography published only after his death, "a flash of passion" passed between the two and they spent the night together. A decade later, Moura would join Wells in London, and while always refusing to marry or live with him, became his companion for the rest of his life. Wells was involved with many remarkable women, but he was drawn to none of them as he was to Moura Budberg.

She went on to establish herself as one of London's most well-connected figures, positioning herself at the centre of a vast social network (incidentally becoming Nick Clegg's great-great-aunt). For Wells, Moura embodied what he described in his autobiography as the "Lover-Shadow" – the dark side of the personality that eludes conscious awareness – and it is true that his encounter with her transformed Wells's view of himself. She told him she could no longer enter the Soviet Union for fear of arrest – to go back would be to risk her freedom, even her life. But when Wells revisited the Soviet Union in 1934 – this time to talk with Stalin – he discovered that Moura had been in the country on at least three occasions in the past year.

Travelling on to her native Estonia, he confronted her with his discovery. To begin with Moura denied everything, but then she told Wells she had been planted on him by the secret police – just as she had been planted on Gorky. She had no alternative, she explained: working for the secret police was the price of life. Wells would not accept that Moura had no alternative. Were there not some things one must never do whatever the circumstances, actions so dishonourable that it would be better to die than commit them? Unmoved by Wells's challenge, Moura laughed and responded with a question of her own. Had he not studied biology? Did he not know that survival was the first law of life? For the species, Wells answered, not the conscious individual. Again Moura laughed, and let the matter go.

Wells's discovery of Moura's hidden life triggered a mental crisis from which he never fully recovered. In an incessant stream of propaganda, he had always insisted that science could be used to construct a new world – along with a higher species to live in it. His scientific romances tell a very different story. When the time traveller journeys into the future, in The Time Machine, he finds a world built on cannibalism, with the delicate Eloi seemingly content to be farmed as food for the brutish Morlocks, and travelling on into the far future finds a darkening Earth where the only life is green slime. In The Island of Dr Moreau the visionary vivisectionist performs vile experiments on animals with the aim of remaking them as humans. The result is the ugly, tormented "beast-folk" – a travesty of humanity.

Wells's fables were a kind of automatic writing – messages from his subliminal self that his conscious mind dismissed. They teach a lesson starkly at odds with the one he spent his life preaching: the advance of knowledge cannot deliver humans from themselves, and if they use science to direct the course of evolution the result will be to engender monsters. This was Wells's true vision, always inwardly denied, and for much of his life expressed only in his scientific romances. Moura released this "esoteric philosophy" (as he later described it) into Wells's conscious awareness.

No longer trusting her, he tried to break with her; but he needed her too much, and was forced to accept that he was not the conscious individual he had imagined. There was no intelligent minority that could seize control of evolution, only a process of unending drift. It was a revelation that left him without hope, a state of mind reflected in one of his last books, The Mind at The End of Its Tether (1945). But that is not the full story, for Moura gave Wells a happiness he had not known before, a mood of serene acceptance expressed in The Happy Turning, also written near the end of his life, where he turns from struggling to change the world to contemplating "the deathless finality of beauty".

Like many others Wells had gone to Soviet Russia in the belief that a new type of humanity was being created there. But the homunculus never materialised – it was an apparition, more insubstantial than the ectoplasm that appeared by sleight of hand in spiritualist seances. Gorky, who believed humans were not far from becoming gods, died miserably, probably poisoned on Stalin's orders. Too weak to write, the god-builder dictated the last entry in his notebook: "The end of the novel the end of the hero the end of the author."

The hopes that led to Lenin's corpse being sealed in a cubist mausoleum have not been surrendered. More than ever, science is seen as a technique for solving the insoluble. Leonid Krasin's failed attempt to preserve Lenin's body has been followed by other projects of technological resurrection – including further attempts at cryonic suspension. A more radical approach is that of the American futurist Ray Kurzweil, who proposes not resurrecting the body but instead shedding it altogether and uploading minds into cyberspace. In The Singularity Is Near: When Humans Transcend Biology, Kurzweil suggests that an explosive acceleration in the growth of scientific knowledge is under way, which will enable people to migrate into a virtual world of their own creation. Already, Kurzweil tells us in Transcend: Nine Steps to Living Well Forever, we can prepare for immortality through a programme of vitamin supplementation, diet, exercise and preventive medical care, which will enhance longevity until it becomes possible for us to leave the flesh behind. For all its hi-tech novelty this is as incredible as the idea that dead scientists are at work to save the living, or a cryogenically preserved Lenin waiting to return to life and revolutionise the world.

The fantasies that possessed the psychical researchers and the god-builders still have us in their grip today. Freezing our bodies or uploading our minds into a supercomputer will not deliver us from ourselves. Wars and revolutions will disturb our frozen remains, while death will stalk us in cyberspace – also a realm of mortal conflict. Science enlarges what humans can do. It cannot reprieve them from being what they are.

John Gray's The Immortalization Commission: Science and the Strange Quest to Cheat Death is published by Allen Lane