In Ray Bradbury’s horror short story, ‘The Next in Line’ (1955), a woman visits the catacombs in Guanajuato, Mexico. Mummified bodies line the walls. Lying awake the next night, haunted by her macabre tour, she finds that her heart ‘was a bellows forever blowing upon a little coal of fear … an ingrown light which her inner eyes stared upon with unwanting fascination’.

Our present era is one in which the heart of culture is blowing hard upon a coal of fear, and the fascination is everywhere. By popular consent, horror has been experiencing what critics feel obliged to label a ‘golden age’. In terms of ticket sales, 2017 was the biggest year in the history of horror cinema, and in 2018, Hereditary and A Quiet Place have been record-breaking successes. In both the United States and the United Kingdom, sales of horror literature are up year over year – an uptick that industry folk partly attribute to the wild popularity of Netflix’s Stranger Things (2016-). And the success isn’t merely commercial. Traditionally a rather maligned genre, these days horror is basking in the glow of critical respectability. As The New York Times remarked this June, horror ‘has never been more bankable and celebrated than it is right now’.

As any historian of the genre will tell you, horror has had previous golden ages. Perhaps ours is just a random quirk of popular taste. But perhaps not. Perhaps we are intoxicated by horror today because the genre is serving a function that others aren’t. Can’t. Horror’s roots run deep, but they twist themselves into forms very modern. The imagination’s conversion of fear into art offers a dark and piercing mirror.

My earliest horror memory is Stay Out of the Basement (1992), one of R L Stine’s Goosebumps series of young adult novels. In the story, a botanist accidently creates a hybrid plant clone of himself. When the clone comes to life, he tries to steal his humanoid self’s life. The botanist’s children unmask the imposter, and in a mess of green blood and plant mush, the clone is felled with an axe. The rescued father disposes of the rest of the mutating plant matter, and the family is all set to live happily ever after. But at the very end, the daughter is standing in the garden and feels a small plant nudging her ankle. The plant whispers to her: ‘Please – help me. I’m your father.’ Stay Out of the Basement is no masterpiece, but I was young, and it struck me cold.

Horror is what anthropologists call biocultural. It is about fears we carry because we are primates with a certain evolved biology: the corruption of the flesh, the loss of our offspring. It is also about fears unique to our sociocultural moment: the potential danger of genetically modifying plants. The first type of fear is universal; the second is more flexible and contextual. Their cold currents meet where all great art does its work, down among the bottomless caves on the seabed of consciousness. Lurking here, a vision of myself paralysed in the dirt, invisible to those I love.

Horror has always been with us. Prehistoric cave paintings are rife with the animal-human hybrids that remain a motif of horror to this day. Every folktale tradition on Earth contains tales of malevolent creatures, petrifying ghosts and graphic violence. The classics are frequently horrifying: in Homer’s Odyssey, when the Cyclops encounters Odysseus’ men, the monster eats them, ‘entrails, flesh and the marrowy bones alike’.

‘During the formation of the human brain, the fear of being grabbed by sharp claws, dragged into a hole and eaten was not an abstraction’

We have always told horror stories, and we always will. Because horror is an artistic expression of an ontological truth: we are creatures formed in no small part by the things to which we are averse. Fear is a base ingredient of consciousness, partaking of brain circuits that are so ancient humans share them with all vertebrate lifeforms. As the neuroscientist Antonio Damasio has described, the whole weird soup of human feeling emerged as a result of our beginning to process whether to ‘approach or avoid … certain places or things or creatures’. Our cognition absorbs reality as a vast spectrum of potential encounters, and horror alchemises the dark end into art.

Thus, evolutionary analyses of horror mention monsters as the genre’s most defining feature. As the philosopher Stephen T Asma puts it, ‘during the formation of the human brain, the fear of being grabbed by sharp claws, dragged into a dark hole and eaten alive was not an abstraction’. For a quarter of a million years – the vast majority of Homo sapiens’ existence as a species – we lived outdoors, with giant hyenas, saber-toothed cats and other carnivores representing a real threat to life. That other ancient health risk, the biological pathogen, manifests itself in the tendency of monsters to be not only violent but also disgusting – feral, oozing blood and saliva, bearing their infectious teeth. From the evolutionary perspective, horror’s vast monstrous menagerie echoes with Paleolithic peril.

Historically, horror’s willingness to play directly to our evolved physiology has seen it earn a low reputation. Western culture was built on a vision of ourselves as above the beasts, above the beastliness of acquiescing helplessly to the demands of the body. But horror can bypass all intellect, extract from us an embarrassingly animalistic response. The skittish physicality of the ‘jump scare’ is a manipulation of what biologists call the startle response, present in all mammals. And cruder horror always contains that other ghastly reminder of our physicality: gore. Gore disgusts us, and the way that gore can be darkly compelling to us disgusts us. Whenever horror is criticised, it is criticised for staging a dark carnival of physicality. Perhaps the only sort of media we moralise more than we do horror is that other mainliner of bodily response, pornography.

Horror’s historical ghettoisation has meant that weightier, smarter horror reliably gets labelled as something else. The finest films of our current golden age have been dubbed ‘elevated horror’ and ‘post-horror’. In literary circles, works of horror seen as sufficiently cerebral get relabelled ‘Gothic’. It’s certainly true that great horror is always about more than gore. But we should be careful not to gentrify the genre by cleansing it of everything but the philosophy.

There are always beings that want to bite us, scratch us, puncture our fragile flesh. There is the terrible old coercion of brute, muscular force, the lethal threat of contagion and infection. There is darkness, disorientation. And looming explicitly or symbolically in all horror is that vast shadow that the anthropologist Ernest Becker said ‘haunts the human animal like nothing else’: death.

‘And he that sat on the cloud thrust in his sickle on the Earth; and the Earth was reaped’. Witness the machinations of that famous slasher, God (Revelation 14:16). Horror encodes the story of our long primate journey, but these biological foundations support the towering edifice of culture. And for millennia, horror merged with our oldest cultural phenomena: religion and folklore. In fact, for most of its history, horror wasn’t really art, as we tend to understand that term today. It certainly wasn’t fiction. Prior to about 1750, in our pivot toward the Enlightenment, the best horror stories can all be found within theology and lore. In Europe, for generations Satan was every bit as petrifying as Pennywise, the murderous clown of Stephen King’s It (1986). Demonic forces were terrifyingly real; in the Bible, Jesus spends almost as much time performing exorcisms as he does healing people. There were widespread societal panics about the threat of werewolves and vampires, and tens of thousands of women were murdered as witches.

This isn’t to judge the credulity of bygone peoples. But the reason that horror – unlike say tragedy, or comedy, or the epic – didn’t exist as an artistic genre until relatively recently is that its deep history is fundamentally pre-scientific. Nothing in the annals of art is as scary as what you’ll find in bygone worldviews. Who needs make-believe scares when everyone you know is awaiting the day of judgment, at which point an angel will sweep a sickle across the Earth and make the blood run for hundreds of miles? It is no coincidence that the Gothic – horror’s regal antecedent – emerged precisely at the moment when lots of people began to believe that God really might be dead. Modern horror is in part the story of what happens when our threatened minds shed a theology. Once holy texts can no longer entirely encode the terrors of being, horror enters fully the arena of art.

However, the old ways cast a long shadow. In the pantheon of genres, horror remains an adolescent, and it has a sort of adolescent relationship with its past: half rebellion, half dependency. On the one hand, more than any other genre, horror loves to thematise the coldest sorts of atheism. ‘All my tales,’ said horror grandee H P Lovecraft, ‘are based on the fundamental premise that common human laws and interests and emotions have no validity or significance in the vast cosmos-at-large.’ In The Silence of the Lambs (1988), amused by what he sees as clear evidence for the absence of any benevolent deity, the charmingly evil Hannibal Lecter ‘collects church collapses, recreationally’.

We turn to horror to explore ‘the biggest and darkest questions’ – good versus evil, the tribulations of the soul, the end of days

On the other hand, horror is marked everywhere by the centuries it spent wedded to otherworldly belief systems. In 2018’s biggest horror movie, Hereditary, an obscure figure from demonology possesses a teenage boy and wreaks death upon his family. Much Japanese horror features yūrei, tormented and enraged spirits denied a smooth passage to the afterlife. Horror was a dark, mutant child of the Enlightenment, and yet it can’t shake its pre-scientific genes. Its penchant for lurid supernaturalism is a big reason why, when it fails, it can so easily seem puerile. The modern, skeptical mind whispers: This is just silly. Haven’t we outgrown all this? On Halloween – a celebration of horror’s pre-artistic forms – children are meant to have the most fun.

Why does horror have this double-edged relationship with its religious and spiritual heritage? Perhaps because, for all its modernity, the sheer scale of theological enquiry still reflects the genre’s ambition. As leading horror author Joe Hill told me, horror is what we turn to when we want to explore ‘the biggest and darkest questions’. And even demoted from dogma to metaphor, the old myths offer a fine way to channel those grand subjects of which horror is so fond: good versus evil, the tribulations of the soul, the end of days. Even though it requires our suspension of disbelief, the paranormal presents us with the very real prospect of brittle reason splintering against the mystery of reality.

‘There’s a sense of uncertainty and potential wrongness underlying most of human existence,’ the Canadian author Gemma Files told me. All of humankind’s great mythic narratives know this, and horror doesn’t let us forget it. At the core of the numinous impulse – that oceanic feeling in which horror was submerged for so many centuries – is the strange certainty that reality is unpredictable and inscrutable, that certain things will forever resist the reach of the human mind. Horror will always share in this sense. It may have fallen from heaven, but it still isn’t entirely of this Earth. Cormac McCarthy’s The Road (2006) centres on a father and son as they wander across a blasted, post-apocalyptic America. The horrors are everywhere: they discover, chained in a basement, ‘a man with his legs gone to the hip and the stumps of them blackened and burnt’. The man is being harvested by cannibals, piece by piece. It is a bleak, fallen world, where the memory of a time when trout swam in the streams shimmers with celestial grace. When the father meets an elderly man, he tells him: ‘There is no God and we are his prophets.’

And so what of today? Horror reverberates with fears Paleolithic and God-fearing, but it is also always reacting to its present moment. And it seems reasonable to perceive any swell in the production and popularity of horror – any dawning of a new golden age – as the expression of a culture that is afraid. ‘In anxious times,’ David Bruckner, director of The Ritual (2017) and other horror movies, told me, ‘people are more likely to turn to horror. If you have an uneasy night at the movie theatre, you are sort of answering the call of your times.’

‘I think we’re living in a nightmare, basically.’ So said horror legend Ramsey Campbell, when I asked him why he thinks horror is flourishing right now. This is one of those things that cooler heads will say is your mind deceiving you. By many objective measures, for many people, life today is better than ever. But horror has never been too worried about culture’s long-term trajectory; it is always fixated on how it could all go badly wrong, any minute now. Horror is steeped in worry; its narratives frequently open with the calm before a terrible storm. And every person connected with horror that I interviewed smelled doom on the breeze.

Horror has always made good use of our deep aversion to what Lovecraft called ‘the oldest and strongest kind of fear’: the unknown. This is one of the ways in which horror (like the folktale) can display a sort of archetypal conservatism. In general terms, the best way to survive a horror setting is to be supremely, boringly sensible: don’t talk to strangers, don’t stay the night in a foreign town, don’t go to the aid of anyone who looks sick, don’t go into that crumbling old building. If a very attractive stranger tries to seduce you, it is almost definitely a trap. Respect tradition, do not commit sacrilege, listen to the advice of elderly locals. At the heart of a lot of horror is a conservative craving for the predictable and the known. The unpleasant atonal dissonance you’ll hear in every horror score reflects, through the collapse of harmony, the disintegration of familiar and comforting patterns out there in the world.

Horror once worried about the weather gods. Now, with climate change, it just worries about the weather

Horror, then, thrives on discombobulation. And today, the discombobulation is everywhere. The causes of the anxiety are scattershot, and you already know them. There are those scientific breakthroughs of the sort that get Silicon Valley execs psyched, but which many others find deeply, opaquely perturbing. Take artificial intelligence, whose rise has seen more and more science fiction turn horrific: ‘One day the AIs are going to look back on us the same way we look at fossil skeletons on the plains of Africa,’ says Nathan, a central character in Ex Machina (2015). And even if the robots don’t vanquish poor old Homo sapiens, other sorts of scientific experimentation might. One of the great horror trends of the 21st century has been the zombie, and in all of the best works of zombie fiction, the immediate cause of the outbreak is the same: biological experimentation gone horribly wrong. The zombie is the gnashing, lunging embodiment of that modern terror, the global pandemic. People might not fear Satan anymore, but they sure as hell fear Ebola.

Outside of the lab, there is that slower method of planetary destruction: climate change. ‘Horror,’ the author Jeff VanderMeer told me, ‘is the beauty of the natural world juxtaposed against the way we destroy those natural systems without understanding them.’ VanderMeer’s Southern Reach trilogy (2014), some of my favourite horror novels of all time, diluted my enjoyment of the UK’s recent heatwave and refused to let me forget that what I was basking in were the convulsions of an aching planet. A biosphere cast brutally off-balance forms the setting for M R Carey’s The Girl With All the Gifts (2014), where humanity has been devastated by a fungal infection. Where horror once worried about the weather gods, it now just worries about the weather. Climate change, meanwhile, is a major cause of mass migrations, potent fuel for what leading critic Leslie Klinger described to me as horror’s historical trend of feeding off ‘the invasion of foreigners into previously stable populations’. At a base level, we are in-groupish creatures. I’ve spent time with rural, paganish communities who enjoy a singsong and are sexually unrepressed; none of them tried to burn me alive in a giant wicker man. But horror says: you never know. Stick to your own.

If all these fears sound selfish, parochial, insular – don’t get the wrong idea. Horror offers a map of the psyche and, like fear itself, is inherently apolitical. It can easily offset its archetypal conservatism with a radical sort of anarchism. Horror might thematise our fear of the unknown – but it also warns about clinging too stubbornly to the familiar. In a lot of horror, survival is predicated on a capacity to quickly adapt to brutal change. Horror has little time for the conservative sentimentality that swirls around ideas such as institutions and tradition, and even something like the nation-state is often revealed as a sort of frilly, doomed illusion. The protections of social hierarchy or private property are never of any use, and horror loves to punish characters who arrogantly believe that wealth will shield them. In horror, the consolations of the past melt in contact with the white-hot heart of present fear. Conservatism fails because it is revealed that at bottom there is nothing to conserve. As the author Michael Marshall Smith put it to me, great horror often declares: ‘It’s just you versus the monster. Always has been. Always will be.’

In this, lots of horror is intensely universalising. Frequently, a scenario comes down to a simple contest between humankind and something else. Splitting up is a suicidal move in horror; survival often follows an impulse toward communal effort. Similarly universalising is the way that, at extreme moments of threat or fear, a given character’s skin colour or gender or nationality will often be effaced. At horror’s pitch, we perceive a simple human, doing what we all do every day of our lives: struggling to live, to persist, to overcome. In The Babadook (2014), a widowed mother is stalked by an amorphous, black-hatted monster that embodies her grief at losing her husband. The monster – her terrible, life-sucking trauma – threatens to claim her son, and destroy what is left of her family. Late in the film, bloodied and exhausted, the mother faces down the Babadook, yelling: ‘If you touch my son again, I’ll fucking kill you.’ The monster is tamed. It is a show of furious bravery that could be any mother, anywhere; courageous love in the face of total disorder.

It’s easy to romanticise horror, but there are also unsexy, functional reasons why it’s having its moment right now. The streaming revolution has given creators a reliable and direct way to reach a dedicated, self-selecting audience. Trusting this audience, distributors like A24 and Blumhouse have put a great amount of both creative wherewithal and cold hard dollars into horror cinema. The genre has always been reliably profit-turning, but it has also always been prone to the lazy recycling of ideas and tropes. Today, even experimental horror can be profitable. In literature, meanwhile, a revival of interest in horror greats like Shirley Jackson, as well as a slew of Stephen King adaptations, has been a boost for the genre at large.

Yet on their own, these tantalising products would never suffice to make horror soar. Horror has been with us since the dawn of storytelling. It manifests the fears of the human animal, and even today echoes the slippery spiritual suspicion that reality isn’t what it seems. Our world is ripe for upending, and horror expresses that best. Horror can thrive today because ours is a strange and febrile cultural moment. It seems every civilisation has believed they are on the brink of cataclysmic change; such an idea has a weird narcissistic appeal. But today there is everywhere a deep feeling that the horses of disaster are about to plunge in the heavy clay. There is a sort of great loop being completed here: as horror has morphed from theology to art, the ruinous power has moved from the judgment of God to the hand of humans. The end of days in one programmer’s idle tinkering, in one laboratory’s overlooked quarantine protocols. Robert Louis Stevenson, author of The Strange Case of Dr Jekyll and Mr Hyde (1886), wrote, ‘Sooner or later, everyone sits down to a banquet of consequences.’ As a genre, horror is forever pulling up a chair, licking its lips at the feast to come.

Breathing this nervous air, current horror – like the theology that provided its former home – is animated by the full spectrum of human psychology. It is driven by our desire to stop all the clocks, shrink into a bubble of the familiar and the known, reject all things foreign. Equally, current horror is shot through with the bone-deep knowledge that if we can’t adapt, we will perish. Its narratives warn us not to cling to outdated consolations, to recognise that we all face the same monsters, in the end. The world has always been dark and full of terrors, and horror has always known it. The dark pleasure of enjoying horror is all about countenancing this awful truth from within a little bubble of safety. It is about the doppelgängered headspace of loathing the real thing but craving its imaginative facsimile. If the genre of horror has a master virtue, a single human quality that it consistently exalts, it is an old one: bravery. We are certain to need that, wherever we are headed.

With thanks to all the horror authors, editors, screenwriters, directors and critics who generously gave me their time to explore horror: Nick Antosca, Stephen T Asma, David Bruckner, Ramsey Campbell, Noël Carroll, Ellen Datlow, Gemma Files, Steven Gerrard, Joe Hill, Carole Johnstone, Leslie Klinger, John Langan, Lisa Morton, Andy Nyman, Jami O’Brien, Xavier Aldana Reyes, Priya Sharma, David J Skal, Michael Marshall Smith, Eugene Thacker, Paul Tremblay, and Jeff VanderMeer.