Over the last two years, I undertook the task of reading as much of the British short story as I humanly could. Invited to prepare a Penguin Book of the British Short Story on a very substantial scale, I made the point to my commissioning editor that anthologies rarely seemed to be the product of much rigorous reading. Quite often one had the impression that the anthology editor’s choice depended on what authors he already knew. Most anthologies seem to have been produced after the editor had read at most 200 stories. In the end, I probably read 20,000.

There was something to be gained by assuming that one actually knew very little about the British short story. First, the known knowns. I wrote down 300 or 400 names of writers of fiction from the last 200 years and investigated their bibliographies. Next, the known unknowns. Library catalogues are peculiarly unhelpful in that they do not often distinguish between a novel and a collection of stories; still, the act of typing “and other stories” into the catalogues of the British Library and the London Library produced an immense number of interesting volumes by authors I knew of, as well as those I did not. (The subsequent task of trying to establish whether the author was British or not was sometimes rather more difficult.) Finally, I decided to read as much short fiction as I could in the context in which it first appeared: in magazines, journals, even newspapers.

AS Byatt reads 'At Hiruharama' by Penelope Fitzgerald Read more

The quantity of short fiction published in this way is insurmountable. Between 1890 and the first world war there were at least 34 magazines publishing short fiction in Britain alone. Some of them, such as the Strand, had huge circulations; others, such as the Yellow Book, have immense literary importance. Journals such as these have been looked at by scholars, if not always by the general reader. Others languish in the darkness of library stacks. The Daily Mail published a large quantity of short fiction in its first decades; nothing I saw was worth resuscitating, but it was clear just how much pleasure it had given to readers on the day of publication.

The inescapable impression, over the course of decades, was the importance of this literary form. Investigation revealed that, before the first world war, journals were happy to pay generously for stories – for a popular writer, the Strand could pay £350 in 1914. This is at a time when the average annual salary of a family doctor was £400, and when (as Arnold Bennett tells us in a brilliant story, “The Matador of the Five Towns”) the weekly wage of a star footballer was fixed at £4 a week. No wonder there was eager competition among writers; no wonder the best writers of the day, including Rudyard Kipling, DH Lawrence, Bennett, Joseph Conrad and HG Wells, placed the form at the very centre of their creative practice.

But, as I read on through the daunting mass of published works, something more surprising became apparent. The short story was certainly at the centre of literary endeavour, and had marked the spot where great white male writers had performed their most astonishing feats of realism, but that was not all. The short story often produced voices from the margins of literary endeavour. Often a periodical might find space for a short story that was extravagantly experimental in style; that was fantastical to the point of the bizarre; that tried something out, in short. This made sense. If, as an editor, you had an established and popular writer with a crowd-pulling lead at the front of an issue, not much would be lost if you also included something truly out of the way towards the back. Wells started out as the supplier of bizarre scientific trifles; quite quickly, he became the writer whose name would sell an issue of a magazine.

Facebook Twitter Pinterest Rudyard Kipling placed short stories at the centre of his creative practice … Photograph: Culture Club/Getty Images

So, reading through the periodicals, one comes across a story, published in the 1850s by Charles Dickens in his editorial capacity, in which a narrator falls through the surface of the Earth to land in an underground territory inhabited entirely by murderous skeletons. Walter Besant, in the 1890s, wrote a story quite out of his usual manner that anticipates the magic realism of Italo Calvino’s The Cloven Viscount by half a century. In my anthology there is a wonderfully unpredictable sea story from the 1840s by Frederick Marryat in which a midshipman falls hopelessly in love with a wind in human form. With the mindset often created by these stories, publishers might take a punt on a novel in a similar vein – Lewis Carroll’s similar underground world, or George MacDonald’s At the Back of the North Wind. The culture of experimentation and fantasy continued well into the 20th century, and you can always find a story about a collision between supernatural powers and public transport, about talking dachshunds, about the events that take place between a man falling out of a fifth-floor window and being killed by the impact, about unreal worlds, about the inner life of a heroin addict.

The marginal and experimental stories, however, are only one way in which the short story reached out beyond its safe, central territory. For almost the entire history of the short story, writers who found themselves disadvantaged in society by birth or nature could interest an editor in a piece of short fiction. Again, nothing much would be lost if it didn’t work. Female writers found that they could be indulged with a short commission, and they took full advantage of the opportunity.

The editors who saw the inherent interest of exotic subjects also saw the virtues of voices from elsewhere, with London editors indulging explorers of Scots and Welsh folktales. Voices that reported with interest and concern on urban minorities, such as Elizabeth Gaskell or Dickens, were succeeded, surprisingly early, by voices that came from those communities themselves. Israel Zangwill and Arthur Morrison, from the 1880s, were publishing sketches specifically about the society they came from in the East End, and, in Zangwill’s case, about Jewish immigrants. Working-class writers, such as Leslie Halward and Jack Common, could also sometimes place a short story; fascinating writers, who were only patchily supported by the literary world. Support could be quickly withdrawn – I think one of the most wonderful writers in the anthology is a Bradford woman, Malachi Whitaker, who published 78 astonishing stories between the late 1920s and 1934 before falling silent for 40 years. But even irregular support enabled something to emerge from obscurity.

Facebook Twitter Pinterest … as did Jean Rhys in the 1920s. Photograph: Bloomsbury

From time to time, short story editors supported writers from even further afield. Jean Rhys was unable to find work as an actor – her Caribbean accent was too strong for the London stage – but could publish short fiction in the 1920s. The Windrush generation discovered a small paternalistic interest in their experiences of London, and what resulted was Samuel Selvon and, on a much grander scale, VS Naipaul. (Naipaul turned the experience of the patronised 1950s author of Caribbean sketches into fiction in the much later Half a Life and Magic Seeds.) Those voices from elsewhere, too, could encompass other minorities. An author wanting to interest or outrage a readership could always resort to tales of sexual minorities – and my anthology includes Henry Fielding’s scandalous lesbian story, “The Female Husband”, as well as Ada Leverson’s “Suggestion”, the drawling monologue of an unmistakably homosexual teenager from the 1890s. Members of sexual minorities, especially gay men and lesbian women, sometimes found their most compelling voices in short fiction – W Somerset Maugham, Sylvia Townsend Warner, GF Green, Rhys Davies, Francis King, Ali Smith and Angus Wilson follow close on each other’s heels in my anthology, and there are plenty more, such as EF Benson, EM Forster, CHB Kitchin, Jackie Kay and Ronald Firbank.

These voices were aided, perhaps, by a point that hadn’t occurred to me – the possibility of immediate topicality. Short stories could address urgent social issues, such as immigration, the sexual revolution in the 1960s, the New Woman in the 1890s, and what women in the 1920s were to do with their lives. They could make immediate use of developments in technology, including trains, the wireless and the internet. It could also address current events. This was a matter of practicality. A novel took time to write and produce; a short story could be written to order, and be in print the same week. Magazines were publishing short stories about life on the battlefield in September 1914. The first of Mollie Panter-Downes’s short stories about the second world war was published a week after war was declared. Literature could be engaged and argumentative, and relevant to the front pages. This is not the whole story, and the best short stories could be gloriously irrelevant – in fact, one of V.S.Pritchett’s greatest short stories, “When My Girl Comes Home” was turned down by the New Yorker on the grounds that its postwar setting was, in 1968, out of date. Nevertheless, much of the excitement of short fiction springs from a shared sense, by writers and the first readers, that this is engaging not just with a recent situation but an evolving one. When Alun Lewis’s superb “Private Jones” was first published in 1942, nobody knew how its hero’s story would end – what his nation’s immediate future held, indeed. The fervour of its closing pages is compelling when you realize that the first British readers would have had no idea what language their rulers would be speaking in a year’s time.

Topicality could be more cryptic than the direct description of current events. It was striking to see how a particular mood or subject started to possess the best short stories. A feeling of deliberate, cruel flippancy comes to the forefront soon after the end of the second world war, as in the work of Angus Wilson. There is a remarkable run of brilliant feminine two-handers in the 1920s and 1930s, not all of them written by women – AE Coppard, an underrated name, provides one of my anthology’s most dazzling in “Olive and Camilla”. As I read, I found I couldn’t escape the sense of growing hysteria in many of the most impressive short stories published before the first world war, often with a demonic edge. The sequence I found myself proposing, without immediately noticing, forms an appalling crescendo. The devil appears in Max Beerbohm, and his servants in MR James; a GK Chesterton story describes exhumation, madness, irrationality with a terrible justification; Saki tells, with every appearance of good humour, what it might be like to plan the murder of every Jew in a country town; and finally Kipling, on the verge of the outbreak of war, provides an incomparable story of the howling madness of crowds, culminating in the House of Commons plunging into hysteria. You can feel something going appallingly wrong, and the writers mapping it all out.

At the end of a systematic period of reading, I had amassed a pile of perhaps 300 short stories. After reading through the collected stories of all the established names, I had forced myself to pick one from each – incredibly hard in the case of varied and accomplished writers such as Elizabeth Taylor and VS Pritchett. Other well-established names I happily discarded altogether. It was amazing how some lazily acclaimed short story writers turned out to be mechanical in the highest degree when read in bulk. As well as those, I had gathered a very large number of short stories that had seemed interesting, startling, absorbing, or simply very beautifully done. It was an unexpected pile. The British short story, as far as I could see, was wildly experimental, predominantly extrovert, relished humour in the most surprising places, and was not very genteel at all. There were moods of trembling, withdrawn sensitivity and privation, but also passages of riot, violence, great stomping fury, hilarity and even hysteria. It came from writers you wouldn’t expect to be allowed to speak, and was often about people that hadn’t been permitted visibility.

Facebook Twitter Pinterest Sherlock Holmes and Doctor Watson in Strand magazine Vol 6.(1893). Photograph: Musuem of London Photograph: Musuem of London/Museum of London

Sometimes, talking to general readers, I was struck by their expectations of what short stories were. They seemed to be in two minds. Some appeared to expect the short story to be intimate, restrained, domestic and full of hints about emotional states running under the surface. It represented an idea of Anton Chekhov, Katherine Mansfield and Dubliners rather than the reality; nevertheless, that seemed to be the impression of the form. Others seemed to expect the short story to be an ingenious sort of puzzle, probably culminating in something called “a twist in the tail” – some kind of revelation that made nonsense of everything up to that point.

By contrast, my idea of the short story was closer to Bennett’s “The Matador of the Five Towns”, in which a visitor to Stoke-on-Trent is abandoned by his host and taken by an acquaintance, a doctor, to see the local newspaper offices, and then to a football match. The star footballer’s wife happens to be in labour; the doctor and the narrator are summoned to the pub that the footballer runs; a tragedy follows, marked by a conversation of truly Shakespearean irony and cruel pettiness. The story contains a huge social panorama, and is very informative about any number of things – how much footballers earn, how newspaper sellers pay for their copies, how pigeons carry the sports results, and so on. It is breathtakingly specific about the world it describes – at the football match, “the gentle, mannerly sound of hand-clapping was a kind of light froth on the surface of the billowy sea of heartfelt applause”.

What I looked for, and what I found in abundance, was a quality of energy. That doesn’t mean noisiness; some of the most compelling stories I found were nothing more than people talking quietly in a room, showing evidence of patient attention, and listening. Nothing much happens in EM Delafield’s “Holiday Group”. An impoverished but quite posh vicar decides, after a small legacy, to take a summer holiday for the first time in years. His wife, Julia, finds a boarding house prepared to take them and their three small children so late in the season – it is quite hard.

A novel took time; short stories could be written to order and address current events

The children behave pretty well and certainly enjoy themselves; Julia can’t stop herself from falling asleep far too early in the evening. At the end, the landlady attempts to overcharge them; they have a lengthy, unreported discussion, and the matter is resolved to everyone’s satisfaction. That’s it. Delafield places the quiet little family so exactly in their social context that we read the pianissimo story with total absorption. The final sentence is: “Her heart swelled with gratitude at the thought of her kind husband, her splendid children, and the wonderful holiday they had all had together.” It is absolutely not ironic. They have had a wonderful holiday, and we are completely glad on their behalf. The first time I read Delafield’s story, I was astonished to find that I had tears in my eyes at the end.

The energy in short stories may come from an understanding of scale. Many of the greatest examples appeared to have some subterranean understanding that they were going to be done within 8,000 words, and from that occasionally sprang a conscious strategy. A writer might decide, like Delafield or Whitaker, that this small space could beautifully reflect a quiet, overlooked moment in an undistinguished life. Or one might take advantage of the restrictions, and strain at the borders of the form in a magnificent story of a political tyranny. If you fill your few thousand words with crowds and violent action, with any number of characters and sweeping landscapes of humanity, then the form will feel pressed, urgent, overpowering – the opposite of leisurely. These stories remain closely observed and specific, like all the best writing, but the detail presses towards a delirious sense – as Vladimir Nabokov once observed with admiration about a John Cheever story – of slightly too much happening in it. But in truth writers can do anything they like with those few thousand words.

This immense variety and energy was created primarily by a conviction among editors that something interesting, unusual and new might find a readership. If it succeeded, the journal would take another story by the same writer, and nurture a particular talent. This process continued until very recently – Horizon encouraged the experimentalist Anna Kavan, and John O’London’s Weekly had a sort of relationship with a marvellous miniaturist, Malachi Whitaker. In time, a publisher might well take the view that the author merited a collection in hard covers. This might not be a very profitable venture in itself. Victor Gollancz, one of the most hardheaded of interwar publishers, took the view that a collection of short stories sold between a quarter and a sixth of an established author’s novels, with a rigid maximum of 4,000. Nevertheless, there was a periodical market, and a prestige market for collections which nurtured all sorts of writing.

It is fair to say that this very successful structure worked well until very recently. These days there are no outlets in the UK that will pay a writer of short stories in a way that will sustain a career. There are outlets that pay tiny amounts, and mass market publications occasionally take a short story from a very well‑established name, paying respectably. The structure of a number of well-paying, well-read journals that would adopt a writer of short stories and develop his or her career in tandem with an appreciative readership has more or less disappeared. The only chance for writers is to be taken up by an American magazine, such as the New Yorker, which has strongly supported the short story careers of Zadie Smith and Tessa Hadley. There is no British publication now that could have done anything comparable – even Granta’s interest in the British short story appears to lapse for years on end.

What has replaced it, and what is often acclaimed as heralding a renaissance in the British short story, is the establishment of any number of competitions. The fledgling writer of short stories no longer submits a piece to an editor who may pay for it; he or she puts it in for a competition. The two primary short story competitions in the UK are the BBC National short story competition and the Sunday Times short story competition. The first prize is £15,000 and £30,000 respectively, which is more than respectable. This benevolent support seems like a good idea. Yet the BBC has never been given to a writer from an ethnic minority. The Sunday Times has only rewarded Junot Diaz and Yiyun Li. Neither prize has ever shortlisted a gay male author, and between them, over the years, only three lesbian writers as far as I can see. This is not an unusual blind spot for literary prizes, including the supposedly “diverse” Man Booker prize. But it contrasts with the vivid and widely acclaimed contribution that sexual, social and ethnic minorities were free to make to the form within an apparently chaotic market economy, in a period when their lives might even be proscribed by law.

Whether or not the blind spots and prejudices of literary juries also result in a preference for what people ought to read over what they will read is an interesting question. It’s certainly true that, looking at recent competition entries, the emphasis changes sharply in comparison to the whole tradition. The British short story has always loved comedy, and its greatest practitioners greatly value humour, even in the most harrowing contexts. That is not something you can expect prize juries to reward, since it’s easier to be articulate about a serious topic than about serious literary expertise.

Facebook Twitter Pinterest Zadie Smith, whose short story career has been strongly supported by magazines. Photograph: AGF/Rex Photograph: AGF s.r.l./REX/Rex

Some of the warmly humane masterpieces in my anthology include Kipling’s “The Village That Voted the Earth Was Flat”, both harrowing and profoundly interested in the terrible cruelty of comedy; brilliant exercises in comedy by Wilkie Collins, PG Wodehouse, Taylor and Elizabeth Bowen and a very characteristic story by Pritchett (for my money the greatest of all British short story writers) which tells a vividly realised episode from a tiny corner of what must be called the human comedy. I wouldn’t have thought any one of them would stand the smallest chance of winning a prize. Instead, they came into being because editors put their faith in them, and because people wanted to read them.

I made a point of telling almost everyone I met that I was working on an anthology of short stories. I wanted to hear what they had to say. Interestingly, the one question almost everybody asked me was this: “When does the short story begin?” It’s difficult to imagine a historian of drama or film being asked this, for different reasons – either a literary form seems contemporary with the human spirit, or to depend on a technological innovation. The short story is rather different, and the question is a good one.

Fiction on a small scale is, from one point of view, as old as the English language. You could claim that there are short stories of a certain sort in the King James Bible and The Anglo-Saxon Chronicle; that short Elizabethan prose romances are at the root of the form. That’s one way of looking at it. Another is to point out that the phrase “short story”, referring to the literary form, dates no further back than the last decades of the 19th century. Probably the truth lies somewhere in between.

Undoubted masterpieces were being written by Heinrich von Kleist in Germany by the beginning of the 19th century. American writers were seen to start a serious practice in the short story by the 1820s at the latest, for a particular reason – the American public preferred cheap, copyright-evading editions of English novels to indigenous products. In the 1830s, the flood of British imports drowned the American novel, and American writers such as Edgar Allan Poe found magazines the chief paying outlets for native fiction. In Britain, short fiction had maintained an interesting, minor presence for some decades.

If the short story format began at a certain point in history, might it – like the epic and the masque – have an end?

The explosion in the popularity of magazines created a hunger for short pieces, bringing news about worlds known and unknown; these started to come in the form of factual pieces, but also fictional explorations of the real world at hand, the remoter parts of the world, and fantastic and incredible worlds that never existed. When Chambers’s Journal first appeared in 1832, its editors promised its readers “lots of nice little stories about travellers in Asia and Africa”. They probably weren’t talking about fiction at that point, but, as one quickly sees, the rising public interest in news from abroad gave immense energy to short fiction. The answer, then, is that the short story as we know it evolved gradually between the 17th century and the middle of the 19th, and was at its peak after the founding of the Strand in 1891, and for decades afterwards.

But the regularity of the question made me wonder whether it was being asked, not about the beginning of a literary form, but about its end. If the short story could be said to begin at a certain point in history, might it – like the epic, the eclogue and the masque – have an end, too? Might that end be approaching?

I hope not. It may simply be that we haven’t fully explored the ways in which a short story might, these days, reach a paying audience. It could work financially for an established author to publish an individual short story as a cheap ebook, but, for a first-time author, publishing a short story as an ebook is simply dropping it into a hole.

Nevertheless, changes in publishing could herald a huge range of new possibilities for short fiction. I don’t think it’s been widely appreciated among authors or editors that, in a world where fiction was published only in virtual form, there would be no problems with length. You could publish Ernest Hemingway’s six-word short story and charge a penny, if you thought anyone would buy it; you could publish a novel of 2 million words without worrying about the costs of production or the discouraging difficulty of holding the thing. Moreover, the costs of production matter much less against the size of the eventual readership.

To cut a long story short Read more

What seems absolutely clear is that the establishment of a readership of whatever size is crucial. Ideally, it should be a paying one. Without that – if a literary form relies exclusively on awards handed out by professionals – the form has no future. At the end of the marvellous reading task I had set myself, I was thrilled by the exuberant, ceaselessly inventive body of work out there. I hope the anthology shows some of that quality, and hints at the way that the short story encompassed the human comedy. If it is given the chance to find an appreciative readership, there is no reason why its energy should be lost.

• Philip Hensher is the editor of The Penguin Book of the British Short Story Vols I &II (Penguin Classics). He will be talking with Tessa Hadley, Adam Mars-Jones and Shena Mackay on the future of the short story on 12 November at Waterstones Piccadilly, London W1J. The Penguin Book of the British Short Story: Volume I and Volume II (Penguin Classics, £25). To order a copy of either for £17.50, go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min. p&p of £1.99.