In 1966, the British Museum acquired a curious object of convoluted provenance and sinister reputation. It is a black disc of highly polished stone, 1.3cm thick and 18.9cm in diameter, with a small handle-like protrusion to one side, pierced with a tiny hole. The thing is plainly a mirror of sorts, small enough to be held in the hand, and carved – so the museum established, after centuries of confusion – from obsidian: the glassy stone from which the Aztecs fashioned their ritual implements. It is usually exhibited alongside a desiccated and much-repaired flat leather case to which are attached two labels. The first, dating from 1842, when the "wondrous speculum" was sold at auction, simply says "Lot 48". The second inscription is from the late 18th century, and likely quickens the hearts of serious scholars and New Age occultists alike: "The Black Stone into which Dr Dee used to call his spirits."

The handwriting belongs to its former owner, Horace Walpole. The politician and collector seems to have chanced upon the enigmatic slab (which he thought was made of marble) in the collection of Lord Frederic Campbell, who was at a loss to say just what it was for. But Walpole was well apprised of the life of John Dee, the Elizabethan astronomer and conjurer who fell in and out of favour with the Queen, and was said to have communed regularly with the angels Gabriel and Raphael. Indeed, an excitable legend concerning his ink-black mirror or "scrying stone" states that it was given to him by one of these angelic interlocutors. The more plausible story has it that Dee bought the stone from a member of the Spanish court, and that it is an Aztec artefact of the late middle ages.

Dee's black mirror does not appear in The Dark Monarch, the current show at Tate St Ives that posits hitherto unheeded links between "magic and modernity" in British art of the past century or so. It has in fact been restored to its historical and cultural context as part of the British Museum's Moctezuma exhibition. But avatars of the mysterious and depthless stone are everywhere in the St Ives show, which seems, like the magician himself, to be in thrall to the spirited seductions of gleaming black objects and darkly reflective planes. It's not simply a dusky Gothicism that seizes on artists as diverse as Barbara Hepworth and Derek Jarman, Graham Sutherland and Cerith Wyn Evans. Nor does the show merely return us to the familiar gloom and swirl of fin-de-siècle mysticism. Instead, it sets up a relay between modern and contemporary art and much older strains of occult thought. The black mirror (which historically fascinated artist and magus in equal measure) is actually and allegorically a portal between the most arcane practices and the mainstream of British painting and sculpture.

As an occult accoutrement, the black mirror has a venerable history. The belief that spirits could be summoned in a dark glass or smooth stone goes back at least as far as Euclid, who writes of images appearing in the space between the viewer and the magic surface into which he peers. By the 14th century, scrying with a black stone or a convex glass mirror was widely held to be a common practice among witches. (In fact, by an odd semantic sleight, the object itself came to be known as "the witch".) So tenacious was the fear of such unholy rituals that in 1318 Pope John XXII published a letter decrying catoptromancy or divination by mirrors, and eight years later issued a bull, Super illius specula, that excommunicated all those engaging in the idolatrous art. During seances in the 16th century, demons were said to have been summoned from the depths of mirrors, and as late as the 18th century it seems the church was still pursuing adepts of the dark glass: an account from 1712 describes the arrest of a catoptromancer at the School of Theology in Paris.

There is something deeply seductive about the image of a pristine surface that frames unspeakable possibilities, so it is not surprising that a small literary canon has grown around the motif of the black mirror. In the extremely eccentric 15th-century Italian dream narrative Hypnerotomachia Poliphili, the protagonist finds himself in an avenue whose marble walls have been inlaid with two perfect circles of jet, so that he is mirrored to infinity. Five centuries later, Truman Capote's short story "Music for Chameleons" seems to unfold in the depths of a black mirror, the "senseless flickerings" of which recall an untuned television. And in his fatally recursive tale "The Mirror of Ink", Jorge Luis Borges makes a black mirror – or rather its common variant, a small pool of ink poured into the hand – the instrument of a terrible revenge.

In such stories, the black surface seems to function as a mise-en-abyme, swallowing among endless reflections those who dare to face it. It's perhaps this aspect of self-reference that several of the artists in The Dark Monarch respond to, alongside the more elemental, occult properties of the black mirror. Jeremy Millar's Assemblage I is an expanded replica of a Sol LeWitt sculpture from the mid-1970s (somewhat resembling a modular display unit) to which have been added certain ritual elements, including an obsidian disc and a hefty clump of fragments of the same material. The self-involved stones seem to leach energy from the surrounding structure, biding their time before deploying their revelatory power. Simon Periton's Mint Snoopier is a convex, oval mirror, elaborately patterned in silver and black and placed ominously above a gallery door. And Eva Rothshchild's sculpture Actualisation – which consists of two glass globes: one black, the other transparent – summons the most familiar image of fairground crystallomancy and at the same time essays an elegant reflection on what is seen and unseen, manifest and occulted.

Other works in the show, less obviously rapt before the mythology of the black mirror as such, nonetheless seem to know something about the visual and metaphoric properties of dark reflective surfaces. Cerith Wyn Evans exhibits a series of inky monochrome photographs, taken by his father, of tree segments that look like the faces of watchful spirits. Gillian Carnegie's painting Black Square turns Kasimir Malevich's modernist icon of the same title into a thickly layered and unsettling treescape. In the hands of these artists, shiny black surfaces suggest not metaphysical voids but darkening landscapes – a sort of homely, rural occultism surmised in the slick black and twilit colours of Clare Woods's painting Daddy Witch.

This intimacy of magic and landscape art is not all that surprising; the black mirror has played a crucial, though now mostly occluded, role in the development of landscape painting in the past two and half centuries, especially in Britain. Tinted glass mirrors, flat or convex, were part of the paraphernalia of the picturesque: that 18th-century mode of seeing which sought to domesticate the more unruly vistas of the natural sublime. Following the example of Claude Lorrain, who was said to have employed tinted mirrors when preparing his paintings, tourists and amateur artists were encouraged, for example, to turn their backs on views of the Lake District and gaze into their "Claude glasses": hand-held black mirrors that now seem like precursors of the digital camera's ubiquitous screen.

By the end of the 18th century, the Claude glass had become a satirical shorthand for the facile touristic gaze. "The effect is unspeakably interesting!" gushes a sightseer in The Lakers, James Plumptre's comic opera of 1798 – but its dark entrapment of the natural world still resonated for Romantic poets. Coleridge wrote of a nature "miniatured on the mind of the beholder, as a landscape on a convex mirror", Wordsworth of the poetic desire to throw over natural objects "a certain colouring of imagination". (By contrast, John Ruskin later railed against the employment of the black mirror, calling it "one of the most pestilent inventions for falsifying nature and degrading art which was ever put into an artist's hand".) The optical and imaginative, if not occult, fascinations of the device survived into the 20th century in such works as Matisse's Anemones with a Black Mirror (1919) and Gerhard Richter's many large mirror paintings.

The Dark Monarch seems to attest to another, peculiarly British, afterlife of the images summoned in the black mirror: one that embodies both occult and art-historical tendencies. The Neo-Romantic painters of the mid-20th century were invested alike in landscape as inspirited territory and object of aesthetic remaking. Paintings by Bryan Wynter and Karl Weschke show a natural world ominously palled in black. Graham Sutherland's Dark Hill and Black Landscape (both from 1940) are at once – as co-curator Michael Bracewell puts it – "pictorial excursions into the consciousness of time and place, tinged by dark prophecy", and reminders of the technologically enhanced Romantic vision of a century and a half before. Sutherland was well acquainted with the mechanics of both ways of seeing: a photograph from 1946 shows him examining his Crucifixion (at St Matthew's Church, Northampton) in a hand-held black mirror.

Among the postwar inheritors of the Neo-Romantic vision was Derek Jarman, whose Super-8 films of the 1970s – notably Journey to Avebury, from 1971 – show him keenly attuned to the spectral remnants of a ritualised English landscape. The Dark Monarch includes Jarman's 1975 film Sulphur – from his series The Art of Mirrors – in which elegant and sinister black-clad figures disport themselves in various ritual attitudes, periodically shining a circular mirror straight at the camera so that it casts everything else into shadow. Jarman was fascinated by Dee; around the time he made Sulphur, he completed a screenplay based on the magician's life, and aspects of this unmade film crept into Jubilee, The Tempest and The Angelic Conversations. Dee's mysterious rituals with the black mirror – an object, wrote Capote, with a "frivolous power" – were for Jarman an abiding image of the alchemy of film-making.

There is, of course, much else to see in The Dark Monarch – an exhibition of its considerable generic and historical scope is not reducible to the adventures of one motif, no matter how seductive. It's an exhibition embedded, for example, in the artistic heritage and milieu of St Ives itself – the title of the show is taken from Sven Berlin's notorious roman à clef of 1962. And it is touched in numerous places by the rough magic of Cornwall. But among its most powerful aspects is the way it reminds us of the darkling strain in familiar artists and works: Barbara Hepworth's Group of Three Magic Stones, Paul Nash's monstrous landscape photographs, a sinister stone mask by Henry Moore. A black seam runs though the show, made of pure obsidian.

The Dark Monarch: Magic & Modernity in British Art is at Tate St Ives until 10 January 2010.