A new exhibition at the Ashmolean Museum in Oxford brings together a multitude of objects and artworks – there’s a “poppet” or rag doll with a stiletto stuck through its face, an amulet containing a human heart, a wisp of “ectoplasm” apparently extruded by a medium in Wales, and too many others to count – from a dark world of nonsense and superstition that we ought to have outgrown a long time ago. At least, that’s how I imagine rationality would view it. I find myself in an awkward position rationality-wise, because my name is listed on the website of the Rationalist Association as a supporter, and at the same time I think this exhibition is full of illuminating things, and the mental world it illustrates is an important – no, an essential part of the life we live. I’d better try to work out what I mean.

I’ll start with William James. In his book The Varieties of Religious Experience (1902), James takes an interesting approach to his subject: he’s not trying to persuade us of the truth of this religion or that, or to unpack some complexities of dogma, or to interpret religious stories for the new 20th century. The book is about what the title says: religious experience – what it feels like to be converted, or to lose one’s faith, or to be in a state of mystical ecstasy, or of existential doubt. James’s examples are drawn from the testimonies of believers and unbelievers alike, and the question of whether there is a God, and whether Jesus Christ is his son, and so forth, is of little interest to James’s main enquiry: only the effects of believing it matter here. For example, we may doubt that the Virgin Mary actually, in fact, physically appeared to Bernadette Soubirous at Lourdes (we may doubt that there ever was a Virgin Mary in the first place) but the vision, or whatever it was, was clearly profoundly meaningful to Bernadette, and her account of it was meaningful to many others, and it certainly had an effect on her and the life she led.

Trying to understand superstition rationally is like trying to pick up something made of wood by using a magnet

And of course she’s not alone. Countless thousands of thoughtful and intelligent people have had experiences of a kind that they call religious. James paid them the compliment of taking these experiences seriously, and produced a classic of psychological insight. But could there be a Varieties of Magical Experience? Could the mental universe that produced witch bottles and sigil, and grimoires, and the whole idea of magic itself, be rich enough to sustain an examination of that sort?

The universe of magic is a large place. It contains phenomena ranging from simple good luck charms to complicated systems of belief and practice such as astrology and alchemy, and it comes to us from prehistory, and from every part of the world, and it still flourishes today. The variety of ideas and objects it contains is almost limitless; the one thing they have in common is that rationalism would scoff at all of them as absurd, outdated, meaningless superstitions that aren’t worth wasting time on.

A human heart in heart-shaped lead and silver case, found concealed in the crypt beneath Christ’s Church, Cork, 12th or 13th century – and now in the Spellbound exhibition. Photograph: © Pitt Rivers Museum, University of Oxford

But rationalism doesn’t make the magical universe go away. Possibly because I earn my living as a writer of fiction, and possibly because it’s just the sensible thing to do, I like to pay attention to everything I come across, including things that evoke the uncanny or the mysterious. Homo sum, humani nihil a me alienum puto (I am human, I consider nothing human alien to me). My attitude to magical things is very much like that attributed to the great physicist Niels Bohr. Asked about the horseshoe that used to hang over the door to his laboratory, he’s claimed to have said that he didn’t believe it worked but he’d been told that it worked whether he believed in it or not. When it comes to belief in lucky charms, or rings engraved with the names of angels, or talismans with magic squares, it’s impossible to defend it and absurd to attack it on rational grounds because it’s not the kind of material on which reason operates. Reason is the wrong tool. Trying to understand superstition rationally is like trying to pick up something made of wood by using a magnet.

I have plenty of superstitions, which are my own and no one else’s (I don’t believe that anyone else would feel more able to write a novel, for example, if they used the only kind of pen and paper that works for me) but one of the interesting things about Spellbound, the Ashmolean exhibition, is that it illustrates beliefs that many people in many places and during many centuries have held in common. Belief in witches, for one thing, is more or less worldwide. In Christian countries it reached a pitch of hysterical panic between the 15th and the late 18th centuries, at a time when tensions between Protestant and Catholic powers were at their highest, and when the medieval world of faith was being challenged by the new thinking of the Enlightenment. Among other things, it was a systematic exercise of cruelty and horror: during this period, writes Malcolm Gaskill in the exhibition catalogue, “there were around 10,000 trials for witchcraft in continental Europe, the British Isles and North American colonies”.

Unlike many human failings, this was not entirely the result of stupidity. Many intelligent people believed that witchcraft existed, and that it was right and proper to stamp it out by killing those who practised it. Nor is that cast of mind safely buried in the past. Until quite recently, people known to be intelligent have felt it was acceptable to put their names to arguments like this: “If we really thought that there were people going about who had sold themselves to the devil and received supernatural powers from him in return and were using these powers to kill their neighbours or drive them mad or bring bad weather – surely we would all agree that if anyone deserved the death penalty, then these filthy quislings did?” That was CS Lewis, in Mere Christianity, 1952.

A ‘poppet’ of stuffed fabric in Edwardian-style black dress with stiletto through face, from 1909–13. Photograph: © The Museum of Wtchcraft and Magic, Boscastle

Whether witches were “filthy quislings” or harmless village healers, they and those who believed in witchcraft and magic existed in a shared mental framework of hidden influences and meanings, of significances and correspondences, whether angelic, diabolic, or natural. Everything in the exhibition testifies to a near-universal belief in the existence of an invisible, imaginary world that could affect human life and be affected in turn by those who knew how to do it; and so do millions of other objects of similar kinds collected, exhibited, studied, or uncollected, unknown, lost, throughout the world and every period of history. As do legends, and ghost stories, and folk tales. If anything is a permanent fact of human nature, this is.

I find it endlessly fascinating, and I call that world “imaginary” not to disparage or belittle it. Imagination is one of our highest faculties, and wherever it appears, however it “bodies forth / The forms of things unknown” (Theseus in A Midsummer Night’s Dream), I want to treat it with respect. At its most intense it becomes a kind of perception, as in William Blake’s notion of “Twofold Vision”, by which he means what we see when we look “not with but through the eye”: the state of mind in which we can “see a World in a Grain of Sand / And a Heaven in a Wild Flower”. Other poets describe something similar: in Wordsworth’s “Ode on Intimations of Immortality from Recollections of Early Childhood” he recalls a time “when meadow, grove, and stream, / The earth, and every common sight, / To me did seem / Apparelled in celestial light, / The glory and the freshness of a dream.” Thomas Traherne’s vision of “orient and immortal wheat” in the everyday corn comes from the same apprehension.

I’m relying on poetry to make this point because I think that poetry itself is a kind of enchantment. The effect that certain lines and images can have on us can’t be explained by translating them into simple modern English. The very form is part of the meaning, and the sound the poem makes works like a spell on our senses and not only on our minds. But it’s not just true of poetry. Everything that touches human life is surrounded by a penumbra of associations, memories, echoes and correspondences that extend far into the unknown. In this way of seeing things, the world is full of tenuous filaments of meaning, and the very worst way of trying to see these shadowy existences is to shine a light on them.

The Witch and the Mandrake (1812) by Henry Fuseli. Photograph: © Pitt Rivers Museum, University of Oxford

This shadow world – the state that Keats called “Negative Capability, that is when a man is capable of being in uncertainties, Mysteries, doubts, without any irritable reaching after fact & reason” – is where the imagination is at home, and so are ghosts and dreams and gods and devils and witches. There, possibilities are unlimited, and nothing is forbidden.

But we have to be clear about what our imagination is. What it isn’t is just a fanciful way of telling a story that isn’t true, or a pretty decoration that we apply to something else to make it attractive, and that isn’t fundamentally important itself. I have a high regard for the scientific writing of Richard Dawkins, but I think that sometimes he expresses a view of the imagination that I simply can’t agree with: “We don’t have to invent wildly implausible stories: we have the joy and excitement of real, scientific investigation and discovery to keep our imaginations in line.” (The Magic of Reality, 2011: my italics). If we have to keep our imaginations in line, it’s because we don’t trust them not to misbehave. What’s more, only scientific investigation can disclose what’s real.

… and a witch trapped in a bottle England, c1850. Photograph: © Pitt Rivers Museum, University of Oxford

On the contrary, I’d rather say that there are times when we have to keep our reason in line. I daresay that the state of Negative Capability, where imagination rules, is in fact where a good deal of scientific discovery begins. In the old expression, reason is a good servant but a bad master, and its powers are limited: no work of art was ever reasoned into existence, for example. David Hume was right: reason is (and should be) the slave of the passions, not their governor. Or as William James put it: “In the metaphysical and religious sphere, articulate reasons are cogent for us only when our inarticulate feelings of reality have already been impressed in favour of the same conclusion.”

The important thing is to be aware of both. Imagination can give us an empathetic understanding of the world of magic; reason reminds us that the cast of mind that persecuted witches is still alive. The Home Office’s ‘hostile environment’ policy appeals to the same dark instinct. The Varieties of Magical Experience still has to be written, as far as I know; and it will only be done successfully by someone who engages the subject with both reason and imagination. Spellbound would be a very good place to start.