Sir James Frazer still casts a long shadow. The wonderful intellectual fugue that is Frazer’s Golden Bough, the 12 volumes of its definitive edition published mostly in the years leading up to the First World War, continues to influence the terms in which great tracts of cultural history are understood – not least in its celebrated genealogy of magic, religion and science. Human culture advances from the magical world-view, explored in such loving (if disapproving) detail by Frazer, towards religion, in which the crudities of magic begin to be purged by moral maturity, en route to the triumph of science.

There is still an assumption in popular writing about religion and science that this is our best way of understanding intellectual history: as a journey from ignorant and inept ways of comprehending how the world works, and how best we manipulate it, towards the objective explanatory scheme of modern scientific analysis. Yet matters are not so simple, as Frazer himself recognised. In practice, “magic” and “religion” as Frazer defines them are inseparably intertwined, to the degree that both assume the existence of invisible agencies that may perhaps be persuaded or coaxed into acting in a particular way. At the same time, magic is more like science in also taking for granted a scheme of things in which effects infallibly follow causes. To this extent at least, “magic has paved the way for science”, says Frazer; and (in the unmistakable voice of Victorian-Edwardian Cambridge) he also argues that it helped to save the world from the tyranny of the uneducated multitude by making a place for the independence and power of the expert – even if this wasn’t the right sort of expertise to win a Trinity prize fellowship in the 1890s.

So problems of definition are likely to be naggingly insistent in The Book of Magic: from Antiquity to the Enlightenment. Brian Copenhaver, a seasoned scholar of late-antique and Renaissance occultism based at the University of California, Los Angeles, in effect refuses to be drawn: following that formidable 20th-century English anthropologist, E E Evans-Pritchard, he settles, more or less, for magic being what “magicians” say it is, rather than pining for a definition of its essence that raises too many questions about evolutionary philosophies or ideological baggage. As such, the selection of absorbing texts in this book covers biblical miracles, Platonic cosmology, alchemy, witchcraft and a good deal more. A single section may include highly technical extracts from Renaissance discussions of natural sympathies and the proper use of the symbolic continuities between the heavens and the earth as well as the prurient and paranoid fantasies of clerical witch-hunters.

This makes for a varied diet but not for much of a clearly organised narrative; and the brief introductory notes to sections and individual extracts are sometimes allusive and telegraphic to the point of being impenetrable. It seems ungracious to question the principles of selection, given the immense range of material that could be drawn upon, and Copenhaver’s exceptional familiarity with it. But I could not quite see why, say, the Latin writer Apuleius is represented with only an extract from his Defence, but no mention of his comic masterpiece, The Golden Ass, one of our best sources for late-antique folklore around magic and witchcraft. Nor is it obvious why a longish extract from Paradise Lost is included, when the same writer’s Comus might tell us more of the younger Milton’s fascination with and anxiety about liberty, nature, magic and sexuality. We have extracts from The Tempest and A Midsummer Night’s Dream but not Macbeth, a far more “realistic” evocation of how Shakespeare’s contemporaries imagined magical practice. And we miss out on some of the recurrent fables of the Middle Ages about great “magicians” – Gerbert of Aurillac (historically a brilliant mathematician who, surprisingly, became pope in 999), the 13th-century Michael Scot (another mathematician) and, of course, Roger Bacon – the Franciscan friar renowned for supposedly creating a speaking oracle in the form of a “brazen head”. Part of the story of magic is to do with what the semi-educated medieval public thought was weird or suspect, from maths to metallurgy.

Copenhaver provocatively includes stories of the healing activities of Jesus as magical narratives; not as odd as it sounds, given that early representations of miracles in Christian art depict Jesus as wielding a kind of magic wand (the American scholar Morton Smith wrote controversial studies on this). But his version of the raising of Lazarus is questionable. John describes Jesus’s reaction to Lazarus’s death using a word that is conventionally translated as “deeply moved”, or even “wound up” or simply “breathing hard”. Copenhaver translates it as “in a frenzy”. Tempting as it might be to think of Jesus as acting in a sort of shamanic trance, this is straining things a bit.

What is clear, though, from the texts included here is that Christianity created a new set of challenges for the family of activities vaguely referred to as magical. On the one hand, the basic Christian narratives assumed the reality of preternatural agents, angels and demons; on the other, they left little or no ground on which you could justify attempts to manipulate spiritual agencies through your own skills. Such agencies were either good or bad. If good (angelic), they would do only the will of the supreme God, so it would be no use trying to get around this for your own ends. If bad (diabolical), they had their own agenda, which was likely to get you into serious trouble. For consistent Christian theologians, then, dealings with these invisible agents were out of the question. “Conjuring”, summoning spirits, was profoundly dangerous.

At the same time, there were all sorts of mysterious interrelations between different bits of the universe which could be exploited for various purposes by means of traditional techniques; alchemy, for instance, though never unequivocally respectable, did not fall under the usual ecclesiastical suspicion of magical invocation. And the deeply rooted assumptions about the natural resonances between different levels of the universe, going back to what were thought of as Pythagorean doctrines, allowed a wide liberty of experimentation. Indeed, one of the many ways in which medieval thought paved the way for what we should recognise as scientific study of the universe was in making this distinction between the manipulating of spiritual agencies and the manipulating of invisible forces; prohibiting the one made more

space for the other.

As these extracts show, the frontier between them was not always so clear, and the prohibitions certainly not ­always observed (there are invocations that are evidently trying to cover the backs of magicians by profuse expressions of reverence for the supreme name or names of God and for the ultimate sovereignty of God). But the way in which a thinker such as the medieval theologian Thomas Aquinas maps out the territory at least creates the possibility of innovative experimentation on natural forces. Aquinas, writes Copenhaver, “takes celestial forces for granted but [restricts] their influence in order to curb naturalism and defend God’s sovereign power”. His discussion of angels in Summa Theologica, excerpted in Copenhaver’s book, is a detailed attempt to distinguish between the natural and the spiritual:

While air holds no shape or colour as long as it stays thin, when condensed it can be shaped and coloured, which is plain to see in clouds, and in this way angels assume bodies from the air . . . Strictly speaking, angels do not talk through their assumed bodies, though there is something resembling speech in that they form sounds in the air that resemble human voices . . . Nor is eating suitable to angels, strictly speaking . . . But the food taken by angels . . . was not really eaten, only symbolic of spiritual eating . . .

A large element in the background of magic was a stubborn puzzlement in the face of natural phenomena that seemed not to be catered for by the ordinary categories of Aristotelian philosophy. Any sort of ­action at a distance – above all, the phenomena of magnetism – required an explanatory scheme rather more capacious than a “billiard ball” materialism, causality as a world of things bumping into each other. Which is why – a nice irony – Newtonian gravity was written off by serious scientific contemporaries as yet another form of magical speculation. It has become something of a commonplace that in the 16th and 17th centuries the boundaries between magic and science were chaotically fluid (the number of founding members of the Royal Society who believed in alchemy or witchcraft was substantial).

Instead of a neat evolutionary progression from ignorant magic to well-informed science, this period brought immense advances in observational accuracy alongside equally ambitious adventures in speculative cosmology, pervasively influenced by Jewish (Kabbalistic) as well as Greek sources. Only gradually during the 17th century did “vitalist”, organic models of natural process give way to atomistic and mechanical structures. And even so, actual scientific advance (as Newton’s theories indicate) was not all a matter of the triumph of such mechanical systems.

There is an obvious contemporary resonance. Modern physics is still, after all, preoccupied with action or causality at a distance; and (although not every scientist or scientific populariser has noticed it) the very notion of materiality – and thus of material causality – has changed beyond recognition in the past century. Not that we are likely to reopen the book on alchemy; but the preoccupations of the “natural philosophers” of the Renaissance do not always seem as remote as they might have done a century ago.

Copenhaver resists generalising about magic; but if we were to be a bit less fastidious, we might want to say that magic typically emerges and flourishes where there is a certain frustration about the refusal of many kinds of public organised religion to solve practical questions. Where public religious practice tends to the contemplative, focusing receptively or adoringly on a depth of reality quite out of human control, magic is entrepreneurial, a private enterprise of trial and error promising high returns for a high degree of risk (spiritual and pragmatic). It is almost universally regarded as politically dangerous for this reason.

Magic appeals to the authority of demonstrable results rather than established hierarchies, even if these results remain promised rather than realised. It is not just a debating point to suggest that certain aspects of modern economic theory work remarkably like ancient and Renaissance magic, appealing to uncertainly charted forces in nature and confidently predicting results that are not always prompt in appearing. If we are baffled by the intellectual respectability of magic in earlier days, it may be illuminating to think about our own willingness to be fascinated (in the Old English sense of bewitched) by ambitious and risk-laden systems based on deeply hidden processes.

To recognise this private and innovative side to magical activity helps make sense of the hostility of both political and religious authority to these practices. Certain kinds of Christianity developed a particular neurosis about certain forms of magic – witchcraft as a matter of covenants with Satan – but the Church was not alone in its severity, as classical histories and ancient Asiatic law codes equally show. Indeed (another irony) the judicial system most consistently sceptical about witchcraft in 16th- and 17th-century Europe was the Inquisition. That appalling 15th-century manual for witchfinders, the Malleus Maleficarum (The Hammer of Witches), was mostly written by an inquisitor who had been disciplined for the wildness of his methods and conclusion:

We conclude that all witchcraft comes from the carnal lust that is insatiable in women, as it says in Proverbs: “There are three things that cannot be satisfied and a fourth that never says ‘enough’” – namely, the “mouth of the vagina” – which is why they gratify their desires even by doing it with demons . . .

It was roundly condemned as dangerous nonsense by the Inquisition authorities. What gave it currency – what allowed it to shape the witch trials of Commonwealth England and Puritan New England in so lethal a way – was its diffusion in printed form. As Copenhaver remarks at one point, print (like the internet today, he might have added) both diffused and confused authority. It could be a tool equally of enlightenment and of murderous bigotry.

Magic is intriguing partly because of its unsettling of authority (a theme wonderfully explored in Susanna Clarke’s Jonathan Strange and Mr Norrell) and its history. We should all like to think that the world is less confined than the masters of our society tell us it is. Yet the ambiguity of the magical enterprise is that it looks for this “something more” of the world in terms of what more our own agendas and desires might attain, what enhanced kinds of control we might acquire. In their diverse ways, both serious public religion and serious scientific research point in a rather different direction: look, listen, be prepared to recognise profound helplessness and ignorance – and in this dispossessed and unsure moment something entirely fresh and unpredicted will be able to flower. As George Herbert wrote, with a mischievous sidelong glance at the frustrated alchemists of his day,

This is the famous stone That turneth all to gold.

Rowan Williams is an Anglican cleric, theologian and poet who was archbishop of Canterbury from 2002 to 2012. He is a lead book reviewer for the New Statesman