Ninety years ago this spring, MR James published one of his most unsettling ghost stories, “A View from a Hill”. It opens on a hot June afternoon, when a Cambridge academic called Fanshawe arrives at the house of his friend Squire Richards, deep in the south-west of England. Richards proposes an evening walk to a nearby hilltop, from where they can “look over the country”. Fanshawe asks if he can borrow some binoculars. After initial hesitation, Richards agrees, and gives Fanshawe a smooth wooden box. It contains, he explains, a pair of unusually heavy field-glasses, made by a local antiquary named Baxter, who died under mysterious circumstances a decade or so earlier. In opening the box, Fanshawe cuts his finger on one corner, drawing blood.

So the two men walk up to the viewpoint, where they stop to survey the “lovely English landscape” spread out beneath them: “green wheat, hedges and pasture-land”, “scattered cottages” and the steam-plume of the last train. The smell of hay is in the air. There are “wild roses on bushes hard by”. It is the pinnacle of the English pastoral.

But then Fanshawe raises the binoculars to his eyes – and that “lovely landscape” is disturbingly disrupted. Viewed through the glasses, a distant wooded hilltop becomes a treeless “grass field”, in which stands a gibbet, from which hangs a body. There is a cart containing other men near to the gibbet. People are moving around on the field. Yet when Fanshawe takes the binoculars from his eyes, the gibbet vanishes and the wood returns. Up, eerie; down, cosy. Up, corpse; down, copse. He explains it away as a trick of the midsummer light.

From there, though, the story takes further sinister turns. The next day Fanshawe bicycles out to Gallows Hill, as it is called locally, to investigate the illusion. In the wood on the hilltop, he becomes convinced that there is someone watching him from the thicket, and “not with any pleasant intent”. Panicking, he flees.

Eventually the grim secret of the binoculars is revealed. Baxter had filled their barrels with a fluid derived by boiling the bones of hanged men, whose bodies he had plundered from the graves on Gallows Hill, formerly a site of execution. In looking through the field-glasses, Fanshawe was “looking through dead men’s eyes”, and summoning violent pasts into visible being. Prospect was a form of retrospect; Baxter’s macabre optics revealed the skull beneath the skin of the English countryside.

We do not seem able to leave MR James (1862–1936) behind. His stories, like the restless dead that haunt them, keep returning to us: re-adapted, reread, freshly frightening for each new era. One reason for this is his mastery of the eerie: that form of fear that is felt first as unease, then as dread, and which is incited by glimpses and tremors rather than outright attack. Horror specialises in confrontation and aggression; the eerie in intimation and aggregation. Its physical consequences tend to be gradual and compound: swarming in the stomach’s pit, the tell-tale prickle of the skin. I find the eerie far more alarming than the horrific: James is one of only two writers (the other being Mark Danielewski) who has caused me to wake myself with my own screaming. Saw sends me to sleep.

A second reason James stays with us is his understanding of landscape – and especially the English landscape – as constituted by uncanny forces, part-buried sufferings and contested ownerships. Landscape, in James, is never a smooth surface or simple stage-set, there to offer picturesque consolations. Rather it is a realm that snags, bites and troubles. He repeatedly invokes the pastoral – that green dream of natural tranquillity and social order – only to traumatise it.

James’s influence, or his example, has rarely been more strongly with us than now. For there is presently apparent, across what might broadly be called landscape culture, a fascination with these Jamesian ideas of unsettlement and displacement. In music, literature, art, film and photography, as well as in new and hybrid forms and media, the English eerie is on the rise. A loose but substantial body of work is emerging that explores the English landscape in terms of its anomalies rather than its continuities, that is sceptical of comfortable notions of “dwelling” and “belonging”, and of the packagings of the past as “heritage”, and that locates itself within a spectred rather than a sceptred isle.

MR James portrays the landscape – especially the English landscape – as constituted by uncanny forces, and part-buried sufferings. Illustration by David McConochie Photograph: The Art Market

Such concerns are not new, but there is a distinctive intensity and variety to their contemporary address. This eerie counter-culture – this occulture – is drawing in experimental film-makers, folk singers, folklorists, academics, avant-garde antiquaries, landscape historians, utopians, collectives, mainstreamers and Arch-Droods alike, in a magnificent mash-up of hauntology, geological sentience and political activism. The hedgerows, fields, ruins, hills and saltings of England have been set seething.

To name names: in music, I think of PJ Harvey’s albums White Chalk (2007) and Let England Shake (2011); of Julian Cope (of course); of the heavily haunted music of Richard Skelton; of English Heretic, the Memory Band, and the Owl Service (whose 2010 album was called The View from a Hill); of Rob St John’s circlings of Pendle Hill in Lancashire, associated with the north-west witchcraft trials and hangings of 1612; of the spectral electronica of Grasscut; and perhaps above all of These New Puritans, especially their albums Hidden (2010) and Field of Reeds (2013), both informed by the estuarine Essex environment in which the band’s members grew up. Their 2010 single “We Want War” thrums with buried furies, encryptions and the stirring of things best left unstirred: “Secret recordings were made in the marsh / I bore a hole in the tree just to see”, chants Jack Barnett, “Knights dance in molecules … / They’re rising back up, they’re rising back up.”

Contemporary literature is rich with this ghost-writing, as we could call it. Paul Kingsnorth’s The Wake (2014) forges an Anglo-Saxon “shadow language” with which to evoke a new-old sense of how in “angland … all is broc”; Melissa Harrison’s novel At Hawthorn Time, published on 23 April, gives ghostly voice to the hedges and lanes of England itself; M John Harrison’s most recent novel, Empty Space: A Haunting (2012), continues his brilliant union of science fiction with everyday English life; and the influence is ubiquitous of both China Miéville’s “weird fictions” and WG Sebald’s wraithy peregrinations along the Suffolk coast and across the Manchester moors, in which landscape is figured as a porous or brittle substance, and in which place operates often only as a sum total of losses and confusions.

Then there is the poetry of Geraldine Monk and Autumn Richardson, Tim Dee’s lyric disinterments in Four Fields (2012), Nina Lyon’s ongoing pursuit of the Green Man myth, Jez Butterworth’s Jerusalem (2009), with its channelling of English place lore into the toking, joking figure of Johnny Byron, and indeed Mark Rylance’s broader commitment to being, as he recently put it, “curious outside the set cosmology”. My book The Old Ways (2012) has four eerie episodes in it, each narrated without analysis or interpretation. One took place at night on Chanctonbury Ring in the South Downs, a wooded hilltop not unlike that on which Fanshawe experiences his malign watcher. I have received more letters and questions about that incident than any other in my books.

In 2013, Mark Fisher and Justin Barton created On Vanishing Land, a 45-minute “audio essay” that traverses the Suffolk coastline from the Felixstowe container port to the Anglo-Saxon burial ground at Sutton Hoo, tracking what Fisher calls “the cumulative forces of the eerie that animate the East Anglian landscape”, and openly in conversation with both the stories of MR James and the music of Brian Eno. There is also, unforgettably, Iain Sinclair’s extraordinary work in prose, film and poetry, committed for decades to the retrieval of heterodox histories and eldritch pasts even as they face occlusion by regeneration and other forms of cultural flattening-out.

In film, I think of Patrick Keiller’s Robinson in Ruins (2010), Michael Hrebeniak’s forthcoming Stirbitch, located around Stourbridge Common in east Cambridge, Ben Rivers’s Two Years at Sea (2011), the mooorland expressionism of Daniel Wolfe’s Catch Me Daddy (2014), and Ben Wheatley’s brilliant A Field in England (2013), in which an alchemist’s assistant joins a group of civil war deserters, high on psychedelic mushrooms, on a hunt for treasure supposedly buried in a field. The film is shot in black and white, which invests the grass, trees, clouds and gun smoke with a strangely luminous intensity of greyscale. The result of the treasure hunt is “unspeakable violence in a rural setting”, in Wheatley’s phrase, or “Apocalypse Now among the hedgerows”, in Jonathan Romney’s. Over the last year I have been working with a young writer and film-maker, Adam Scovell, to adapt a co-written book called Holloway into a nine-minute Super-8 short, inspired in part by Derek Jarman’s early silent film Journey to Avebury (1971). Holloway is about about time-loops and unexplained occurrences in a sunken Dorset lane; among its histories is the hanging at Dorchester in 1594 of four Catholic recusants. In the longer term I want to adapt Algernon Blackwood’s 1907 ghost novella “The Willows” for screen, relocating it from the willow deltas of the Danube to the Phragmites reed forests of the East Anglian fens.

The Duchess of Cambridge looks at Self Portrait as a Drowned Man by Jeremy Millar during a visit to Turner Contemporary, Margate. Photograph: Suzanne Plunkett/Getty Images

The eerie is widespread in contemporary English art, there in the work of (among others) Tacita Dean, Marcus Coates, Steve Dilworth, Stanley Donwood and Jeremy Millar. In 2011, also inspired by Blackwood, Millar created a sculpture entitled Self-Portrait of a Drowned Man (The Willows). He cast his own body in silicone, dressed it in his own clothes, then gouged “his” face and skull with odd puncture wounds, as occurs in Blackwood’s novella. The disconcertingly lifelike (deathlike) “drowned man” that resulted was displayed prone on the gallery floor. It was first shown at Glasgow’s CCA, and proved so unnerving to audiences that warnings had to be issued. It is presently on show at Turner Contemporary in Margate. Earlier this month, the Duchess of Cambridge visited the gallery and was photographed looking at Millar’s pseudo-corpse selfie. The image went viral: two divergent Englishnesses were embodied and juxtaposed, the monarchical/hereditary and the uncanny/unsettled.

Yes, the contemporary eerie feeds off its earlier counterparts, as with Millar off Blackwood, Fisher off James and Scovell off Jarman. A renewed interest in classics of the tradition is in evidence: director Robin Hardy’s 2013 “print” of The Wicker Man (1973) for instance, or Witchfinder General (1968), a film that, as Sinclair put it last year, “really takes on landscape to reveal the underlying sense of psychotic breakdown”, by staging an “argument between the English pastoral … and unscripted brutal violence”. Wheatley cites it, with Culloden (1964), as hugely influential for A Field in England. The radical histories of Kevin Brownlow and Andrew Mollo’s Winstanley (1975) are taken up again by Keiller in Robinson in Ruins, which examines moments of failed revolution in the quiet fields of today’s Cotswolds.

There is also the recent reissue of Alfred Watkins’s cult book of landscape mysticism, The Old Straight Track (1925), the argument of which is largely built around the views from hills, and the totemic status among contemporary place-writers of JA Baker’s The Peregrine (1967) – all bloody killing, field-haunting and ritual reperformance. Vital to the modern moment, too, are the novels of Alan Garner and Susan Cooper; especially Garner’s The Weirdstone of Brisingamen (1960) and The Owl Service (1967), and Cooper’s dazzling The Dark Is Rising sequence, published between 1965 and 1977. Once read, these novels are hard to forget. They lodge and loom in the memory. Garner turned eighty80 last autumn, and a volume of essays exploring his legacy, called First Light, is being compiled at present, with contributions from Philip Pullman, Ali Smith and Neil Gaiman among others. I regard the second book of Cooper’s sequence as among the eeriest texts I know; Helen Macdonald is another for whom Cooper’s novels have been imaginatively vital.

It would be easy to dismiss all this as an excess of hokey woo-woo; a surge of something-in-the-woodshed rustic gothic. But engaging with the eerie emphatically doesn’t mean believing in ghosts. Few of the practitioners named here would endorse earth mysteries or ectoplasm. What is under way, across a broad spectrum of culture, is an attempt to account for the turbulence of England in the era of late capitalism. The supernatural and paranormal have always been means of figuring powers that cannot otherwise find visible expression. Contemporary anxieties and dissents are here being reassembled and re-presented as spectres, shadows or monsters: our noun monster, indeed, shares an etymology with our verb to demonstrate, meaning to show or reveal (with a largely lost sense of omen or portent).

We are, certainly, very far from “nature writing”, whatever that once was, and into a mutated cultural terrain that includes the weird and the punk as well as the attentive and the devotional. Among the shared landmarks of this terrain are ruins, fields, pits, fringes, relics, buried objects, hilltops, falcons, demons and deep pasts. In much of this work, suppressed forces pulse and flicker beneath the ground and within the air (capital, oil, energy, violence, state power, surveillance), waiting to erupt or to condense.

A suitable place for violence? Orford Ness, Suffolk. Photograph: Photograph by Emma Johnson/Getty Images/Flickr RM

Taken together, in all its variety, this work suggests what the writer and archaeologist Eddie Procter recently called a “new landscape aesthetic”: dedicated to a busting of the bucolic, a puncturing of the pastoral. Last month an interdisciplinary conference was convened in Cambridge on “The Alchemical Landscape”, prompted by the “increasing number of writers, artists and film-makers [who] are reinvesting the landscape with esoteric and mythic imagery”, and seeking to assess how this body of work “articulates pressing contemporary concerns”.

What are those pressing concerns, though, and what are the sources of this unsettlement? Clearly, the recent rise of the eerie coincides with a phase of severe environmental damage. In England, this has not taken the form of sudden catastrophe, but rather a slow grinding away of species and of subtlety. The result, as James Riley notes, is “a landscape constituted more actively by what is missing than by what is present”. This awareness of absence is expressing itself both in terms of a vengeful nature (a return of the repressed) and as delicate catalogues of losses (Skelton and Richardson’s poetry, for instance).

There is a dissenting left politics also at work here. Joe Kennedy has sharply documented how the “pastoral has become … a cornerstone of austerity politics” under David Cameron, with the “compensatory cant of Green Toryism” helping kitschly to reduce the rural to Bake Off tents in flower meadows, Hunter wellies, and Keep-Calm-and-Holiday-at-Home messages, even as fiscal forces churn and ruck the landscape. For Kennedy, this “cupcakification” of the nation – this rise of “Mumfordland” – has generated a vigorous anti-pastoral counter-culture, animated by what he nicely calls “the terror in the terroir”. In a blistering 2013 essay, Nick Groom used the infamous “Let’s discuss over country supper soon” text from Rebekah Brooks to Cameron, sent on the eve his leader’s speech to the Conservative Party Conference, to excoriate the Tory reduction of the rural to a Cotswold pageant of G&Ts and predistressed timber tables. Groom deplored the “pastoral cliches that have tyrannised the land for decades”; we need “to start rethinking what we understand by the English countryside”, he wrote, “and we need to be doing it now”. That political “rethinking” is partly apparent in the preoccupation of eerie culture with the English civil war (there in A Field in England, in the band name of These New Puritans, in Keiller’s Robinson in Ruins), a historical moment in which, as Wheatley puts it, the country was “so radicalised that normal people were forced into political positions they had never been in before”.

Digging down to reveal the hidden content of the under-earth is another trope of the eerie: what is discovered is almost always a version of capital. Keiller’s Robinson tracks the buried cables and gas-pipes of Oxfordshire, following them as postmodern leylines, and tracing them outwards to hidden global structures of financial ownership. Wheatley’s deserters rapaciously extract “treasure” from the soil, by means of enslavement and male violence. In his cult novel Cyclonopedia (2008), the Iranian philosopher Reza Negarestani figured oil as a sentient entity, developing Marx’s implication that capital possesses emergent and self-willed properties, that it is somehow wild. Negarestani, like Miéville, is inspired by the weird fiction of HP Lovecraft (1890–1937). I eagerly await the English oil-horror film that must surely soon be shot: Quadrilla as Godzilla, sink-holes as maws from out of which rises Chthulu (Chthoilu). Perhaps filming is already under way, somewhere in a field in Dorset, with nodding donkeys backlit on the skyline.

“Why does the countryside seem such a fit place for violence?” Matthew Sweet asked Wheatley on Radio 3, reflecting on the charnel-house ending of his film. Wheatley, who grew up in Essex, spoke of his long-standing sense of the English countryside as somewhere in which violence was always imminent. “There was something in the land-scape,” he said, “that plainly terrified me ... If you went out into it you could just be killed.” His comments echo those of Sinclair, who invoked King Lear on the heath: “You’re turned outdoors into something more savage than you are, and you know something terrible is going to happen.”

An image from Ingrid Pollard’s Pastoral Interlude series. Photograph: ingridpollard.com

They also uncannily recall those of the photographer Ingrid Pollard, whose remarkable 1988 series of images, Pastoral Interlude, explored the black British experience of the English countryside and especially the Lake District, where Pollard “wandered lonely as a Black face in a sea of white”. “A visit to the countryside,” she wrote in the caption to one image, “is always accompanied by a feeling of unease, dread.” Pollard’s 19th-century photographic techniques allude to the connections between the high-Victorian pastoral and the colonial control in Africa and the Caribbean that enabled that pastoral’s maintenance. The photographer Marc Atkins and the poet Rod Mengham are together making a study of English arable land, responding to fields “that all look the same to some extent”, but that have undergone “utterly different specific histories”, from battlefields to prisoner of war camps to sites of execution.

Contemporary eerie culture is also drawn to the military and security infrastructure that occupies much of England’s land and air space, from Salisbury Plain to Otterburn to Foulness. This dispersed geography of conflict and surveillance has attracted the interests of Garner (Alderley Edge), Fisher and Barton at Bawdsey Manor (the Suffolk home of radar research in the 1940s), Keiller (passim), and WG Sebald at Orford Ness, where still standing are the high towers and fine string-antennae of Cobra Mist, a top-secret cold war over-the-horizon radar station, which allegedly provided returns so clear it could track the movements of individual trains in Siberia. The monumental era of 20th-century detection technology, when structures needed to be vast in order to see further, has proved especially attractive. Tacita Dean and Brian Dillon have both responded to the sound mirrors of Denge, near Dungeness, in Kent. These forerunners of radar are huge “acoustic mirrors”, resembling “listening ears”, designed to concentrate into audibility those forms of distant noise (approaching aircraft engines, for instance) that would otherwise subtend the range of human hearing.

It isn’t hard to see why contemporary eerie culture should be drawn to such evidence of record and detection. If the eerie is – as Fanshawe found on Gallows Hill – about the experience of being watched by a presence that you cannot perceive, then this, certainly, is another cause for its present relevance. For the state has never before been as able to detect and follow the movements of its subjects. Nor – since Snowden – have we been as conscious of the extent to which we are continuously being observed by unseen forces, not always operating with what Fanshawe called “pleasant intent”. Yet state surveillance is no longer testified to in the landscape by giant edifices. Instead it is mostly carried out in by software programs running on computers housed in ordinary-looking government buildings, its sources and effects – like all eerie phenomena – glimpsed but never confronted.

Shortly after A View from a Hill appeared in the London Mercury in May 1925, MR James was contacted by the poet AE Housman, a friendly acquaintance and fellow Cambridge don. Housman admired the story, but felt there was “something wrong with the optics”. It was a nitpick on Housman’s part – he was suggesting that anyone looking through liquid-filled binocular barrels would experience a blurred refraction of vision, rather than its strange sharpening. His literalism missed the point entirely, of course – and it is tempting to read Housman’s quibble with the story as a broader objection to James’s unsettling of the pastoral, a mode in which Housman was deeply invested. For there was, in fact, something very right with James’s optics. Where Housman looked backwards, to the “land of lost content”, James looked forwards, and saw the English countryside not only as a place of beauty, calm and succour, but also as a green and deeply unpleasant land.