John Lennon urged us: "Imagine there's no heaven/It's easy if you try/No hell below us/Above us only sky." Yet the religious aren't turning to Lennonism any faster than Leninism. Today, according to a new book by Lisa Miller, Newsweek's religion correspondent, 81 per cent of Americans and 51 per cent of Brits say they believe in heaven – an increase of 10 per cent since a decade ago. Of those, 71 per cent say it is "an actual place". Indeed, 43 per cent believe their pets – cats, rats, and snakes – are headed into the hereafter with them to be stroked for eternity. So why can't humans get over the Pearly Gates?

In reality, the heaven you think you're headed to – a reunion with your relatives in the light – is a very recent invention, only a little older than Goldman Sachs. Most of the believers in heaven across history would find it unrecognisable. Miller's book, Heaven: Our Enduring Fascination with the Afterlife, teases out the strange history of heaven – and shows it's not what you think.

Heaven is constantly shifting shape because it is a history of subconscious human longings. Show me your heaven, and I'll show you what's lacking in your life. The desert-dwellers who wrote the Bible and the Koran lived in thirst – so their heavens were forever running with rivers and fountains and springs. African-American slaves believed they were headed for a heaven where "the first would be last, and the last would be first" – so they would be the free men dominating white slaves. Today's Islamist suicide-bombers live in a society starved of sex, so their heaven is a 72-virgin gang-bang. Emily Dickinson wrote: " 'Heaven' – is what I cannot Reach!/The Apple on the Tree/Provided it do hopeless – hang/That – 'Heaven' is – to Me!"

We know precisely when this story of projecting our lack into the sky began: 165BC, patented by the ancient Jews. Until then, heaven – shamayim – was the home of God and his angels. Occasionally God descended from it to give orders and indulge in a little light smiting, but there was a strict no-dead-people door policy. Humans didn't get in, and they didn't expect to. The best you could hope for was for your bones to be buried with your people in a shared tomb and for your story to carry on through your descendants. It was a realistic, humanistic approach to death. You go, but your people live on.

So how did the idea of heaven – as a perfect place where God lives and where you end up if you live right – rupture this reality? The different components had been floating around "in the atmosphere of Jerusalem, looking for a home", as Miller puts it, for a while. The Greeks believed there was an eternal soul that ascended when you die. The Zoroastrians believed you would be judged in the end-time for your actions on earth. The Jews believed in an almighty Yahweh.

But it took a big bloody bang to fuse them. In the run up to heaven's invention, the Jews were engaged in a long civil war over whether to open up to the Greeks and their commerce or to remain sealed away, insular and pure. With no winner in sight, King Antiochus got fed up. He invaded and tried to wipe out the Jewish religion entirely, replacing it with worship of Zeus. The Jews saw all that was most sacred to them shattered: they were ordered to sacrifice swine before a statue of Zeus that now dominated their Temple. The Jews who refused were hacked down in the streets.

Many young men fled into the hills of Palestine to stage a guerrilla assault – now remembered as the Hanukkah story. The old Jewish tale about how you continue after you die was itself dying: your bones couldn't be gathered by your ancestors anymore with so many Jews scattered and on the run. So suddenly death took on a new terror. Was this it? Were all these lives ending forever, for nothing? One of the young fighters – known to history only as Daniel – announced that the martyred Jews would receive a great reward. "Many of those who sleep in the dust shall awake, some to everlasting life, and some to shame and everlasting contempt," he wrote and launched us on the road to the best-selling 1990s trash 90 Minutes in Heaven. Daniel's idea was wildly successful. Within a century, most Jews believed in heaven, and the idea has never died.

But while the key components of heaven were in place, it was not the kumbaya holiday camp it has become today. It was a place where you and God and the angels sat – but Jesus warned "there is no marriage in heaven". You didn't join your relatives. It was you and God and eternal prayer. It was paradise, but not as we know it.

Even some atheists regard heaven as one of the least-harmful religious ideas: a soothing blanket to press onto the brow of the bereaved. But its primary function for centuries was as a tool of control and intimidation. The Vatican, for example, declared it had a monopoly on St Peter's VIP list – and only those who obeyed their every command and paid them vast sums for Get-Out-of-Hell-Free cards would get them and their children onto it. The afterlife was a means of tyrannising people in this life. This use of heaven as a bludgeon long outlasted the Protestant Reformation. Miller points out that in Puritan New England, heaven was not primarily a comfort but rather "a way to impose discipline in this life."

It continues. Look at Margaret Toscano, a sixth-generation Mormon who was a fanatical follower of Joseph Smith in her youth. Then she studied feminism at university. She came back to her community and argued that women ought to be allowed to become priests. The Mormon authorities – the people who denied black people had souls until 1976 – ordered her to recant, and said if she didn't, she wouldn't go to heaven with the rest of her family. She refused. Now her devastated sisters believe they won't see her in the afterlife.

Worse still, the promise of heaven is used as an incentive for people to commit atrocities. I have seen this in practice: I've interviewed wannabe suicide bombers from London to Gaza to Syria, and they all launched into reveries about the orgy they will embark on in the clouds. Similarly, I was once sent – as my own personal purgatory – undercover on the Christian Coalition Solidarity tour of Israel. As we stood at Megido, the site described in the Book of Revelation as the launchpad for the apocalypse, they bragged that hundreds of thousands of Arabs would soon be slaughtered there while George Bush and his friends are raptured to heaven as a reward for leading the Arabs to their deaths. Heaven can be an inducement to horror.

Yet there is an unthinking "respect" automatically accorded to religious ideas that throttles our ability to think clearly about these questions. Miller's book – after being a useful exposition of these ideas – swiftly turns itself into a depressing illustration of this. She describes herself as a "professional sceptic", but she is, in fact, professionally credulous. Instead of trying to tease out what these fantasies of an afterlife reveal about her interviewees, she quizzes everyone about their heaven as if she is planning to write a Lonely Planet guide to the area, demanding more and more intricate details. She only just stops short of demanding to know what the carpeting will be like. But she never asks the most basic questions: where's your evidence? Where are you getting these ideas from? These questions are considered obvious when we are asking about any set of ideas, except when it comes to religion, when they are considered to be a slap in the face.

Of course there's plenty of proof that the idea of heaven can be comforting, or beautiful – but that doesn't make it true. The difference between wishful thinking and fact-seeking is something most six-year-olds can grasp, yet Miller – and, it seems, the heaven-believing majority – refuse it here. Yes, I would like to see my dead friends and relatives again. I also would like there to be world peace, a million dollars in my current account, and for Matt Damon to ask me to marry him. If I took my longing as proof they were going to happen, you'd think I was deranged.

"Rationalist questions are not helpful," announces one of her interviewees – a professor at Harvard, no less. This seems to be Miller's view too. She stresses that to believe in heaven you have to make "a leap of faith" – but in what other field in life do we abandon all need for evidence? Why do it in one so crucial to your whole sense of existence? And if you are going to "leap" beyond proof, why leap to the Christian heaven? Why not convince yourself you are going to live after death in Narnia, or Middle Earth, for which there is as much evidence? She doesn't explain: her arguments dissolve into a feel-good New Age drizzle.

True, Miller does cast a quick eye over the only "evidence" that believers in heaven offer – the testimonies of people who have had near-death experiences. According to the medical journal The Lancet, between 9 per cent and 18 per cent of people who have been near death report entering a tunnel, seeing a bright light, and so on. Dinesh D'Souza, in his preposterous book Life After Death, presents this as "proof" for heaven. But in fact there are clear scientific explanations. As the brain shuts down, it is the peripheral vision that goes first, giving the impression of a tunnel. The centre of your vision is what remains, giving the impression of a bright light. Indeed, as Miller concedes: "Virtually all the features of [a near-death experience] – the sense of moving through a tunnel, an 'out of body' feeling, spiritual awe, visual hallucinations, and intense memories – can be reproduced with a stiff dose of ketamine, a horse tranquilliser frequently used as a party drug." Is a stoner teenager in a K-hole in contact with God and on a day-trip to heaven? Should the religious be dropping horse dope on Sundays? But Miller soon runs scared from the sceptical implications of this, offering the false balance of finding one very odd scientist who says that these experiences could point beyond life – without any proof at all.

But even if you set aside the absence of even the tiniest thread of evidence, there is a great conceptual hole at the heart of heaven – one that has gnawed at even its fondest believers. After a while, wouldn't it be excruciatingly dull? When you live in the desert, a spring seems like paradise. But when you have had the spring for a thousand years, won't you be sick of it? Heaven is, in George Orwell's words, an attempt to "produce a perfect society by an endless continuation of something that had only been valuable because it was temporary". Take away the contrast, and heaven becomes hell.

And yet, and yet ... of course I understand why so many people want to believe in heaven, even now, even in the face of all the evidence, and all reason. It is a way – however futilely – of trying to escape the awful emptiness of death. As Philip Larkin put it: "Not to be here/Not to be anywhere/And soon; nothing more terrible, nothing more true". To die. To rot. To be nothing. We wouldn't be sane if we didn't seek a way to leap off this conveyor-belt heading towards a cliff.

So yes, there is pain in seeing the truth about Heaven – but there is also a liberation in seeing beyond the childhood myths of our species. In The Epic of Gilgamesh, written in Babylon 4,000 years ago, the eponymous hero travels into the gardens of the gods in an attempt to discover the secret of eternal life. His guide tells him the secret – there is no secret. This is it. This is all we're going to get. This life. This time. Once. "Enjoy your life," the goddess Siduri tells him. "Love the child who holds you by the hand, and give your wife pleasure in your embrace." It's Lennon's dream, four millennia ahead of schedule: above us, only sky. Gilgamesh returns to the world and lives more intensely and truly and deeply than before, knowing there is no celestial after-party and no forever. After all this time, can't we finally follow Gilgamesh to a world beyond heaven?

To follow Johann's Twitter updates, go to www.twitter.com/johannhari101