“The short story is enjoying a powerful renaissance”, ran a headline in the Spectator in September last year. “After decades of neglect,” it added, “the genre is very much back in fashion.” This isn’t true, but when it comes to short stories fake news is ubiquitous.

Other recent announcements of the short-story renaissance include one in 2014, when the Daily Telegraph called it “the perfect literary form for the 21st century” because brevity suits our dwindling attention span (more on the stupidity of that argument later); in 2013, when the short-story specialists Alice Munro and Lydia Davis won the Nobel and the Man Booker International Prizes, respectively; and in 2012, which Bloomsbury proclaimed “the year of the short story”, publishing five collections in as many months.

It is often said that publishers don’t like short stories because they don’t sell: it’s assumed this proves that readers don’t like them either. Yet, rather than accept the genre as a minority interest, there is always someone – a journalist, a prize jury, a publisher – announcing its comeback.

While bitter experience has shown poetry exactly where it stands in the marketplace, and the novel has shrugged off multiple reports of its death and maintained pre-eminence, the short story is continually characterised as the neglected form that will be great again. The funny thing is, when you explore its history you find the perception of a distant golden age, an undistinguished present and a return to glory has always been around: the short story has a problem with reality.

“The ’nineties,” as H G Wells wrote in the preface to his collection The Country of the Blind (1911), “was a good and stimulating period for a short-story writer.” Thanks to the range of journals available and the quality of their editorship, he believed, “No short story of the slightest distinction went for long unrecognised . . . Short stories broke out everywhere.”

By 1911 things were different. Kipling had gone off the boil (he hadn’t, in fact, but that’s another argument); so had Max Beerbohm and Henry James. Only Joseph Conrad, Wells thought, was producing work equal to his pre-1900 output, but this wasn’t enough to stop the “recession of enthusiasm” for the short story.

At the end of his 1941 study The Modern Short Story, H E Bates predicted that short fiction would be the “essential medium” of the war and its aftermath. In a 1962 article he admitted his mistake, and in the preface to a 1972 reissue of The Modern Short Story he wrote: “My prophecy as to the ­probability of a new golden age of the short story, such as we had on both sides of the Atlantic in the 1920s and 1930s was . . . dismally unfulfilled . . . Even before the war in England the little magazines to which writers of my generation contributed . . . were already dead or dying.” And dolefully he concluded, “This then is the situation of the short story today; if it is not quite one of unmitigated gloom it is certainly not bright.”

Yet that same year Christopher Dolley, in The Second Penguin Book of English Short Stories, noted that, “far from continuing its supposed decline, the short story is enjoying a revival all the more encouraging when viewed against the gloom surrounding the future of the literary novel”. Was Bates merely wrong or reactionary? It appears not.

The avant-garde author B S Johnson, said his collaborator, Zulfikar Ghose, conceived the 1964 collection Statement Against Corpses in response to the “wretched state” into which the English short story had fallen. The pair saw it as “our destiny to revive the form”.

In 2004, in an essay about (what else?) the renaissance of the short story, William Boyd remembered that:

When I published my first collection of stories, On the Yankee Station, in 1981, many British publishers routinely brought out short-story collections. Not any more. Moreover, there was a small but stable marketplace where a story could be sold. A short-story writer could place his or her work in all manner of outlets. The stories in my first collection, for example, had been published in Punch, Company, London Magazine, the Literary Review and Mayfair, and had been broadcast on the BBC . . . Today, in the UK especially, it has never been harder to get a short story published. The outlets available to a young writer that I benefited from in the 1980s have virtually dried up.

And yet Boyd identifies a new enthusiasm for the short story, primarily because of the boom in postgrad creative writing courses, whose workshop model well suits their composition and analysis.

Leaving aside the contradiction between the desolation of Bates’s postwar period and the thriving 1980s scene Boyd remembers, the number of magazines that paid writers for stories peaked between the 1890s and the 1930s. If you were prodigious enough during this period, it was entirely possible to earn a living from short stories. Never­theless, the authors such as Arthur Conan Doyle and F Scott Fitzgerald, who might earn the modern-day equivalent of tens of thousands of pounds for a single story, were always outliers. As Philip Hensher notes in the introduction to his Penguin Book of the British Short Story (2015), what magazines were paying for stories in the late 1880s had barely changed by the 1930s.

If discussions of the short story’s reception lead us into boggy ground, so do attempts to define precisely what the short story is. In his introduction to the impressive Cambridge History of the English Short Story, the first single-volume study of its type, the editor, Dominic Head, avoids doing so, and this is very much par for the course. In his 1991 essay “On Defining Short Stories”, Allan H Pasco wrote that those few critics who devote time to the short story “hedge on definitions, origins, major traits, on just about everything having to do with the short story as a genre”.

William H Gass, proposing one of my ­favourite definitions, proceeds by exclusion before moving into abstraction: “It is not a character sketch, a mouse-trap, an epiphany, a slice of suburban life. It is the flowering of a symbol centre. It is a poem grafted on to sturdier stock.”

In the Cambridge History, Ailsa Cox inadvertently coins a workable, albeit squarely economic, definition when she describes contemporary short fiction as “the least lucrative form of literary endeavour, apart from poetry”. Gerri Kimber, discussing the difference between story, novella and novel, says the difficulty lies with each form using the same techniques. Yet uncertainty needn’t be a bad thing: blurred boundaries can offer greater possibilities. Richard Ford considers it “a relief to observe how many disparate pieces of writing can be persuasively called short stories, how formally underdefined the short story still is in the minds and hands of writers”.

The uncertainty about what the short story is extends to when it began. Boccaccio lurks somewhere in the background, as do Chaucer and anecdote-laden jest books of the Elizabethan era. Some anthologists have gone back to the Old Testament and called the Books of Jonah and Ruth short stories, but these, with oral tales and passages from Homer, represent the form’s prehistory.

The short story as we understand it today is a 19th-century development. “We all came out from under Gogol’s ‘Overcoat’” – a statement that has been attributed to both Turgenev and Dostoevsky – is where Frank O’Connor begins his highly influential 1963 study, The Lonely Voice. Walter Allen, however, in The Short Story in English (1981), identifies Walter Scott’s “Two Drovers”, published 15 years before “The Overcoat”, in 1827, as the first modern short story. Elizabeth Bowen, in her 1936 introduction to The Faber Book of Modern Short Stories, doesn’t go any further back than Maupassant and Chekhov, because, in her opinion, no one else has had such a powerful effect on the form’s development.

Maupassant, taught by Flaubert, brought an extreme objectivity and immediacy to the short story. Chekhov’s great innovation was to promote atmosphere above plot. His stories are less about what happens than how it is told; as Somerset Maugham jokingly said, “If you try to tell one of his stories there is nothing to tell.” Chekhov employs implication and melancholy to mysterious yet profound ends, and although James Joyce claimed not to have read him before he wrote Dubliners (published in 1914, but mostly written ten years earlier), the similarities in technique are striking. And to English and Irish readers, still, it is the stories in Dubliners – with their moments of epiphany, in which characters suddenly see themselves with all illusions stripped away – that define what is most commonly thought of as a short story.

There are undoubtedly skills that set you in good stead as a story writer, not least compression: it is logical that the short form should appeal most to those with the ability to say a lot in a short space of time (or to say a lot without saying much at all, as Raymond Carver achieved when he was edited by Gordon Lish). Beyond that, there are so many directions a writer can take. Most mainstream stories can be traced back to Chekhov or Maupassant, but not the postmodern provocations of Donald Barthelme, or the fable-like conundrums of Kafka, or the subverted fairy tales of Angela Carter, the thought experiments of Lydia Davis, nor even Alice Munro’s domestic Gothic. Perhaps it’s best to keep the definition simple, as John Barth does: short-story writers incline to see how much they can leave out, novelists to see how much they can leave in.

Edgar Allan Poe was even more practical than Barth in his 1842 review of Nathaniel Hawthorne’s Twice-Told Tales. There he described a short story as a piece of work intended to be read in one sitting of up to an hour. Simplistic, perhaps – but it works, and explains why the short story is anything but the perfect form for a short attention span (a myth that often accompanies the renaissance narrative). In 2010, for instance, Neil Gaiman said short stories were “a wonderful length for our generation . . . perfect . . . for your iPad, your Kindle or your phone”.

What does this even mean? Given the need for a piece to be read at a single sitting – say, half an hour for the average New Yorker story – and the compression that demands constant and close attention to the text, it is bizarre to talk up the short story’s suitability for time-poor readers. War and Peace is enormously long but its chapters are short, taking five or ten minutes to read. It also includes a list of characters and, as Flaubert pointed out, Tolstoy often repeats himself. There’s a book for a crappy attention span.

It is understandable but unfortunate that the Cambridge History limits itself to fiction from the British isles and former colonies. Various contributors mention Chekhov and Maupassant, but the book’s focus doesn’t allow their centrality to the development of the short story to be established properly. Katherine Mansfield is discussed in the context of modernism and post-colonialism, but her huge debt to Chekhov, and the part she played in extending his influence to a subsequent generation of writers, is not. Other writers suffer from compartmentalisation: it feels old-fashioned to address the work of Hanif Kureishi and Zadie Smith primarily in the context of multiculturalism. The author of the chapter on this, Abigail Ward, issues a sort of apology for the term, but it would have been better to explore their work in wider contexts.

In its defence, the book covers enormous ground – colonial stories, rural stories, queer stories, comic stories – and makes room for obscure writers beside the heavyweights. There are flaws to compartmentalisation, yet how else to avoid incoherence when the history of the short story, wherever it begins, rapidly fragments into concurrent histories cutting separate channels? At least, with this approach, an expert writes each chapter. Highlights include Heather Ingman on the Irish short story and Roger Luckhurst on weird fiction, that amorphous zone between horror, fantasy and surrealism. Luckhurst and Ingman are excellent guides: able, as several of their fellow contributors are not, to give a strong flavour of individual writers’ styles while situating them within a theoretical framework.

Given the wealth of material available, it is a shame that so much discussion of the short story is infected with ill-informed debate about its popularity. It would be much more valuable to discuss the writing, which encompasses some of the greatest fiction in English: “The Signal-Man” by Dickens; “The Dead” by Joyce; Katherine Mansfield’s “At the Bay”; Kipling’s “Mrs Bathurst”; J G Ballard’s “My Dream of Flying to Wake Island”; “The Company of Wolves” by Angela Carter. These, regardless of genre, are essential reading.

Quality, however, has little to do with popularity. The short story is and will remain a minority interest. This isn’t a defeatist position: if more weight were given to the work, and less to its popularity, some valuable stability could be established. Today, in qualitative terms, the short story is healthier in Ireland than in the UK, and yet there are good young writers out there, working with the form because it suits the stories they have to tell, not because it promises fame and financial reward. The renaissance is not under way and Nell Zink’s advice will be sound for a long time to come:

Don’t write short stories and poems unless you have a trust fund. No matter how perfect they are, no matter what prestigious magazine publishes them, each one will be 200 pages too short to pay the rent.

Chris Power’s story collection, “Mothers”, will be published in 2018 by Faber & Faber

The Cambridge History of the English Short Story

Edited by Dominic Head

Cambridge University Press, 657pp, £99.99