He always comes on his own, this bachelor of antiquarian tastes. Sometimes he is a book dealer, more often an academic. He is a dry, crotchety character, not particularly sympathetic. He is usually on holiday, in East Anglia or an old town in France or Denmark. He is staying in an inn or a hotel, an uncongenial sort of place far from his familiar institutional comforts. In fact he is way out of his comfort zone. And then it begins … the tapping at the window or the rustling or the tangling of the bedsheets.

We know what we are in for, just as surely as we do when we open an Agatha Christie or an Elmore Leonard. The formula is simple, repeated with variations in most of M.R. James’s 33 ghost stories, and still guaranteed to give pleasure today just as it did to those fuddled dons and sleepy schoolboys who first heard James read them by the light of a single candle in the provost’s lodgings at King’s College, Cambridge, or in his last years, as provost of Eton.

It may seem heartless or unsporting to deconstruct these little tales, for the author made no very exalted claim for them. ‘If any of them succeed in causing their readers to feel pleasantly uncomfortable when walking along a solitary road at nightfall, or sitting over a dying fire in the small hours, my purpose in writing them will have been attained.’ The excuse must be that James himself was eager to unpack the formula in the prefaces and occasional articles he wrote on the subject. He was happy to share his own ideas on how a ghost story should be laid out if it was to be effective. ‘The setting should be fairly familiar and the majority of the characters and their talk such as you may meet or hear any day.’ But ‘a slight haze of distance is desirable.’ Unlike the detective story, the ghost story should not be too up-to-date. ‘Thirty years ago’ or ‘Not long before the war’ were proper openings. Close enough in time, therefore, for the reader to think: ‘If I’m not very careful, something of this kind may happen to me.’ The ghost ought to be a contemporary of the person who sees it, just as Hamlet’s father and Jacob Marley were.

The important thing was atmosphere. The setting had to be carefully prepared and evoked. James does this particularly well with the coastal landscape of Suffolk around Aldeburgh – Peter Grimes country. Then it is time for the ‘nicely managed crescendo’. And then it is time for bed. No doubt it is partly because of the proximity of bed (it is from his own bedroom that he has emerged with his spidery manuscript and the single candle) and the arrival in the morning of the ominously named ‘bedder’ that the provost is insistent on one thing: there must be no sex. James tells us more than once that sex is a ‘fatal mistake’ in ghost stories. It spoils the whole business. ‘Sex is tiresome enough in the novels … as the backbone of a ghost story, I have no patience with it.’

And yet, as Darryl Jones, a specialist in horror fiction at Trinity College, Dublin, points out in his first-rate introduction to this compendium of all James’s ghost stories, the phantoms themselves repeatedly break the rules he has laid down for them. What happens, after all, in the typical James story? The bachelor don or antiquary discovers a lost manuscript or artefact – a whistle or an inscription on stained glass – which unleashes supernatural forces that shake his comfortable assumptions about reality. What we can’t help noticing is that these supernatural forces tend to be female. In the build-up, by contrast, scarcely a woman appears, except as a comic rustic or tiresome servant in the inn. But the ghosts themselves are so often women, spurned or murdered or guilt-ridden: Mrs Mothersole in ‘The Ash Tree’, Ann Clark in ‘Martin’s Close’, Theodosia Bryan in ‘A Neighbour’s Landmark’ and the terrible figure in ‘a shapeless sort of blackened sun-bonnet’ in ‘Wailing Well’. We don’t need to have read any of the Freud which James would have run several miles from to interpret what Mr Dunning in ‘Casting the Runes’ finds when he puts his hand into the well-known nook under his pillow: ‘What he touched was, according to his account, a mouth, with teeth, and with hair about it, and, he declares, not the mouth of a human being.’

Jones detects a vagina dentata, and it is hard to dodge the sexual vibrations here or in the other slimy, clutching, intensely hairy phantasms which finger and stroke the beleaguered single man. The spectral bedsheet in ‘Oh, Whistle and I’ll Come to You, My Lad’ attempts to thrust its ‘intensely horrible face of crumpled linen’ close into the face of Professor Parkins as he cries for help. The gender of the bedsheet is unclear, but the fear of domesticity and the fear of sex, tangled and intertwined like the bedsheet, are unmistakable. And even when the ghosts and their victims are both male, the erotic overtones still hum. How horrified James would have been to find ‘Lost Hearts’ or ‘The Residence at Whitminster’ included in an anthology of gay ghost stories, but both would certainly deserve their place, especially the latter.

How much was James aware of the vibrations? How far did he intend them? ‘Reticence,’ he said, ‘may be an elderly doctrine to preach, yet from the artistic point of view I am sure it is a sound one.’ But the reticence he practised went far deeper than the stories he tossed off in the intervals of his vast scholarly endeavours. It went all the way down. His friends thought him as sexless as an angel.

Montague Rhodes James (1862-1936) lived the unexamined life in spades, in the way that only an unmarried don of his generation could. The son of an Anglican vicar of evangelical inclinations, he won a scholarship to Eton, then another to Henry VI’s sister foundation, King’s College, Cambridge, at that time still largely an Etonian preserve. He spent the next 36 years at King’s, as successively undergraduate, dean, tutor and provost, as well as being director of the Fitzwilliam Museum for 15 years. He catalogued the manuscripts in all the college libraries as well as several other great collections and became the world’s leading scholar on Apocryphal literature. He then returned to Eton for the last 18 years of his life as provost. He had thus spent his entire adult existence in ‘Henry’s holy shade’, untroubled by marriage, poverty or anything much else.

Even among his fellow dons his insulation was thought remarkable. His best friend, A.C. Benson, declared that Monty’s ‘mind is the mind of a nice child – he hates and fears all problems, all speculation, all originality or novelty of view. His spirit is both timid and unadventurous.’ James’s knowledge, he conceded, was ‘extraordinarily accurate and minute’ but mainly concerned with unimportant matters. ‘No one alive knows so much or so little worth knowing.’ His mind had nothing constructive in it. ‘He seems to me to be an almost perfect instance of high talent: a perfect second-rate man.’ Benson himself was tormented by his homosexuality as well as by frustrated or delayed ambitions and he was subject to terrible mental collapses. While master of Magdalene he spent more than two years in a nursing home (amazing to think that such a tortured soul wrote the words of ‘Land of Hope and Glory’). He could not forgive Monty’s insouciance or his unthinking resistance to change.

Among the people and things that James hated and refused to countenance at King’s or Eton were: T.H. Huxley (‘a coarse 19th-century stinks man’), Henry Sidgwick and his philosophy, the German higher criticism, anthropology and comparative mythography, Aldous Huxley, James Joyce, Bertrand Russell, J.B.S. Haldane and John Maynard Keynes (for being a renegade Eton-and-King’s man who thought the college needed shaking up). Lytton Strachey returned James’s contempt: ‘It’s odd that the provost of Eton should still be aged 16. A life without a jolt.’

The only modern innovations that I can discover James adopting were the rear-driven safety cycle and the Dunlop pneumatic tyre, which between them enabled him and his friends to cover huge distances on their Continental excursions to forgotten cathedrals and unknown libraries. Even his old school tutor H.E. Luxmoore was shocked by the frivolity of James and his circle. After spending Christmas at King’s one year, he complained: ‘O how the time goes in talk, talk, talk and overmuch eating.’

Last night Monty James read us a new story of most blood-curdling character, after which those played animal grab who did not mind having their clothes torn to pieces and their hands nailscored. The cleverness and gaiety of them all is wonderful and yet if it goes on like this in term time – and it does – where is the strenuous life, and the search for truth and for knowledge that one looks for at College? Chaff and extravagant fancy and mimicry and camaraderie and groups that gather and dissolve first in this room and then in that, like the midges that dance their rings in the sunshine, ought to be only the fringe of life and I doubt if here it does not cover the whole, or nearly so.

James himself did not resist this impeachment. In his sixties, he ruminated: ‘Do you know, I have written an immense deal of stuff and find myself almost incurably frivolous.’

That may be why his phantoms do not, on the whole, haunt. They lie doggo in their holes and only when roused make a single surprise appearance. They are cabaret turns, not implacable disturbers of the peace. The morning after, normality returns. Only in James’s very last published story, ‘A Vignette’, is the first-person narrator repeatedly haunted by the horrifying white face that appears at the hole in the gate of the fir plantation. Indeed, I haven’t noticed the word ‘haunted’ used much anywhere else in the stories. James declared himself dissatisfied with ‘A Vignette’ as ‘short and ill written’. On the contrary, I find it one of his few genuinely disturbing tales. Partly because the scene is based on his childhood home at Livermere, Suffolk, partly because it has a first-person narrator, and partly just because it has no plot, there is a genuine worrying quality to it, an ambiguity for which the word ‘haunting’ fits.

The OED derives ‘haunt’ from the French ‘hanter’, which it says is ‘of uncertain origin’. But Dauzat’s Dictionnaire Etymologique has no such doubts, declaring firmly that it comes from the same root as ‘habiter’. So haunting is a matter of habit; frequency is of its essence. The ghostly connection comes later: for the French, ‘maison hantée’ is a 19th-century Anglicism. In English, the ghostly connection is at least as old as Shakespeare. Richard II tells us that some kings are ‘haunted by the ghosts they have deposed.’ Ghosts then were regular revenants, resident nuisances. In James, they are vestigial; they may not outlast the memory of those that knew them in life. The grandmother in ‘An Evening’s Entertainment’ tells her grandson that ‘maybe things like that do die out in the course of time.’ At the end of ‘A Vignette’, the narrator asks the almost elegiac question: ‘Are there here and there sequestered places which some curious creatures still frequent, whom once on a time anybody could see and speak to as they went about on their daily occasions, whereas now only at rare intervals in a series of years does one cross their paths and become aware of them?’

If ghosts are on the way out, does the same apply to other supernatural things, perhaps even to the Christian faith which so sustained James and the rituals of which were the heart and soul of his daily existence? That is not a question which James would have asked himself or permitted his audience to ask. His whole life can be seen as a series of defensive operations: to protect King’s against the stinks men, to protect the Christian Church against critique and to protect himself against reality. At times, he reproached himself with indolence. But he was never indolent when he was defending his own certainties.

Towards the end of his life, in 1929, James wrote a survey of ghost stories for the Bookman, beginning with Lope de Vega in the 17th century and culminating with Dickens and, his favourite, Sheridan Le Fanu. The article ends in a characteristically self-deprecating but also unusually abrupt way: ‘There need not be any peroration to a series of rather disjointed reflections. I will only ask the reader to believe that, though I have not hitherto mentioned it, I have read The Turn of the Screw.’ Deafening is the only word for the silence. Why will he say no more? What does James think of his namesake’s ghost story, published in serial form in Collier’s Weekly in 1898, just at the time when he was writing his first ‘Antiquary’ stories? Does he admire its deliberately unresolved ambiguities? Does he hate its sensuous implications of ill-defined evil? Or both or neither? Is Henry’s exhaustive psychological analysis precisely what Monty will not permit himself, even if he were capable of it? Like Bertie Wooster, M.R. James seems not entirely at home with the psychology of the individual; for that sort of stuff, he relies on Jeeves. On his only recorded meeting with the other James, he thought that Henry looked ‘like a respectable butler’.

Curiously, both writers claim to be free of responsibility for their creations. Monty declares that ‘I have not been possessed by the austere sense of the responsibility of authorship which is demanded of the writer of fiction in this generation.’ Henry speaks in his own preface of ‘this perfectly independent and irresponsible little fiction’. But they mean quite different things. Monty wants to reassure us that his tales are harmless trifles which a housemaster may safely read to his pupils after they have finished their prep. Henry, by contrast, is warning us that he cannot be responsible for where his little story may end up; it is likely to take us into some dark places from which we may not emerge unscathed.

Many megabytes have been expended on trying to resolve whether Peter Quint and Miss Jessel are ‘real’ or whether they are the creations of the new governess’s highly strung brain. It seems probable, to put it no higher, that the author means to keep us in suspense on this as on many other questions. But what is certain is that the ghosts in The Turn of the Screw are pervasive and persistent presences, and that the damage they do is cumulative. That is what haunting means. Real ghosts, like real art, are always irresponsible. They do not go away when they are told to. That is why they are best kept out of the provost’s lodge.