Two years in the making, our list of the 100 greatest English-language novels of all time is now complete. Having endured many sleepless nights in its compilation, Robert McCrum reflects on who got left out, and why

In the parlour game called “Humiliation”, in David Lodge’s 70s campus novel Changing Places, the players score points by confessing the famous works of literature they have never read. In a memorable comic climax, ambitious academic Howard Ringbaum admits he has never read Hamlet, instantly wrecking his career.

Lodge’s insight into the practice of literature is that everyone who steps into the world of books and letters risks humiliation. Rightly, for the well-being of culture and society, this is a competitive affair. Beneath the eye of eternity, it’s a matter of life and death: either some kind of literary afterlife or (more likely) oblivion.

Facebook Twitter Pinterest Jane Austen’s Emma, published 1815 (No 7).

For the past two years, as I’ve compiled my 100 Greatest Novels list for the Observer and theguardian.com, the fate of the classics has been my special subject. For one enraged online critic, the series was simply “an elaborate headstone for a defunct way of thinking about literature”. Never mind; not only was there a vigorous, sometimes splenetic, discussion of the list and its choices by a dedicated core of well-read correspondents, there was also a surge in subscribers. Some weeks, with titles such as Frankenstein (No 8) or Heart of Darkness (No 32), our virtual audience soared into hundreds of thousands. In total, between 1 and 2 million readers have interacted with the series through the Guardian website.

So: what is a classic? There are many duelling definitions. TS Eliot, Ezra Pound, Italo Calvino and Sainte-Beuve have all written at length on the subject. Calvino’s definition – “a classic is a book that has never finished what it wants to say” – is probably the sweetest, followed by Pound’s identification of “a certain eternal and irresponsible freshness”. One necessary, but not sufficient, characteristic of a classic is that it should remain in print.

Thereafter, the issue becomes subjective. Classics, for some, are books we know we should have read, but have not. For others, classics are simply the book we have read obsessively, many times over, and can quote from. The ordinary reader instinctively knows what he or she believes to be a classic. While our preferences inevitably reflect gender, nationality, class, and education, there is no accounting for taste.

Joseph Conrad, author of Heart of Darkness (No 32), arriving in New York in 1923. Photograph: Bettmann/Corbis

Speaking of taste, I selected, where possible, the title most central to the author’s voice and vision, which is not necessarily the most famous. Consider Jane Austen. Pride and Prejudice is much loved. Northanger Abbey is highly entertaining. Some readers revere Sense and Sensibility. But I chose Emma.

Casual browsers sometimes took a moment to grasp that my list of “top novels” was: a) derived exclusively from fiction written in the English language; b) strictly chronological; and c) gave each writer equal space, an especially restrictive criterion. With a prolific writer, the selection of one “classic” text became almost intolerable. To cite an unfair example – Dickens (No 15, David Copperfield) and Wilde (No 27, The Picture of Dorian Gray) appear in the list on the same footing, with one novel apiece. With such rules, every thoughtful person must concede that any list is bound to have its ridiculous side.

Facebook Twitter Pinterest F Scott Fitzgerald’s The Great Gatsby, published 1925 (No 51).

As the series unfolded, the more its true character as a mission impossible began to take shape. For example, these novels span about three centuries, roughly 1700 to 2000. Compiling a list for the first 100 years was relatively straightforward, from 1800 to 1900 progressively more difficult, and from 1900 to 2000 (my arbitrary cut-off) perilously close to impossible. Here’s why.

In the century that witnessed the making of the English novel, the genre was almost exclusively the work of upper- or middle-class English writers, predominantly male, and often with private means, living in the British Isles. Their novels were addressed to an elite minority, and expressed the concerns of a particular society.

At this distance, in a way that’s quite unthinkable in our own time, a decade – sometimes more than a decade – in which only one novel of classic status appears looks less like a literary crisis than an assertion of good taste. Between The Life and Opinions of Tristram Shandy, Gentleman (1759) and Emma (1815), I did not choose a single competitive title. After 1818, another creative hiatus; then the Victorian novel roars into view, with a newly independent America in the picture for the first time. Now, judging the relative merits of 19th-century fiction, a once simple set of choices became exceedingly awkward.

Facebook Twitter Pinterest Samuel Clemens, aka Mark Twain, author of The Adventures of Huckleberry Finn (No 23). Photograph: Bettmann/Corbis

How to compare The Way We Live Now (No 22) with The Adventures of Huckleberry Finn (No 23) with Kidnapped (No 24)? Was it right to exclude Uncle Tom’s Cabin (hugely popular and influential), in favour of Little Women (also bestselling)? It was essential to acknowledge the emerging American literary tradition, but who trumps whom? Wilkie Collins (The Moonstone) was in, but James Fenimore Cooper (The Last of the Mohicans) did not make the cut. Was there a transatlantic bias? As Man Booker prize judges will know, this issue does not go away.

George Eliot’s Middlemarch, published 1871-72 (No 21).

Then along comes modernism. Long before Conrad, Lawrence and Joyce, the authors and publishers of the three-decker novel were in trouble. Some just gave up. Others found new kinds of employment. In the 1920s, after the cataclysm of the first world war, a new literary landscape emerged and the novel would never be the same again. The idea of “the common reader”, which had sustained a century of book publishing, was doomed.

Our list reflects the trauma behind this transition. I ignored Galsworthy and Bennett, but included HG Wells, dropped Chesterton but kept Conan Doyle. Later, as the list drew to a close, with every choice at a premium, I cursed the leniency I had exercised towards the novels published between 1880 and 1930. For instance, does Jack London (No 35, The Call of the Wild) better deserve a place than, say, James Salter (Light Years and All That Is), Thomas Pynchon (Gravity’s Rainbow) or JG Ballard (Crash), three contemporary writers who got left out?

I love Frederick Rolfe’s Hadrian The Seventh (No 37), and Sister Carrie by Theodore Dreiser (No 33) has a dark energy that’s crucial to the evolution of the American novel. But A Confederacy of Dunces by John Kennedy Toole should have been included, and so should Slaughterhouse Five (Kurt Vonnegut), plus All the Pretty Horses (or perhaps Blood Meridian) by Cormac McCarthy.

So: a few howlers, several regrets, and many sleepless nights. How on earth did I overlook RK Narayan (Swami and Friends), Flannery O’Connor (Wise Blood) or Rose Macaulay (The Towers of Trebizond)? Or, more painful still, Nancy Mitford (The Pursuit of Love) and Shirley Hazzard (The Transit of Venus)?

Facebook Twitter Pinterest DH Lawrence, author of The Rainbow (No 43), circa 1920. Photograph: Bettmann/Corbis

Two years spent with the classics has been a rare opportunity to get close to an occasionally majestic art form. What, if anything, does this list teach? Somerset Maugham, characteristically sardonic, once observed that “There are just three rules for writing a novel. Unfortunately, no one knows what they are.”

Actually, the only rule is that there are no rules. The novel can be about a little girl falling down a rabbit hole, or the obsessive hunt for a whale, or the quotidian wanderings of a Jewish, middle-aged advertising canvasser across Dublin. The free range of our imagination is the only departure lounge in flights of the mind for which, mercifully, there’s no GPS. The reader remains a free agent. In his or her freedom lies the joy of reading: it is unpoliced, solitary and mostly private in a world without frontiers or governments or bosses.

The novel, which began in extremis (Bunyan in prison; Defoe dodging creditors), but learned to entertain, and make money (Sterne cashing in as a bestseller; Dickens selling like a box set), still follows two basic imperatives: obsession or entertainment. PG Wodehouse once said that there are two ways of writing: one is “a sort of musical comedy without music and ignoring real life altogether; the other is going right deep down into real life and not caring a damn.” The creative fault line of “musical comedy” v “real life” never goes away.

Facebook Twitter Pinterest Emily Bronte’s Wuthering Heights, published 1847 (No 13).

There are other continuities. I would argue that not one of these 100 books or their authors is indifferent to the demands of story. With our stone-age brains, we are still a storytelling species. The narrative gene is part of our DNA. As EM Forster put it, in Aspects of the Novel: “Yes – oh, dear, yes – the novel tells a story.”

And yes – oh, dear, yes – I left out some great storytellers: H Rider Haggard (King Solomon’s Mines); L Frank Baum (The Wizard of Oz); Margaret Mitchell (Gone With the Wind); Daphne du Maurier (Rebecca); Ian Fleming (Casino Royale); Charles Portis (True Grit); Erica Jong (Fear of Flying); John Le Carré (Tinker, Tailor, Soldier, Spy), and many other favourites, no doubt.

There were some other deliberate omissions: Elizabeth Gaskell (North and South), whose appeal I confess I’ve never understood; Nathanael West (Miss Lonelyhearts); Rosamond Lehmann (The Weather in the Streets); Norman Mailer (The Executioner’s Song), who was too much of an American celebrity in thrall to the media to be a truly great novelist; Kingsley Amis (Lucky Jim); John Fowles (The French Lieutenant’s Woman), whose work has not worn well; Beryl Bainbridge (The Bottle Factory Outing), a minor masterpiece, but not a classic to equal the other great novels of the 1970s; and Iris Murdoch (The Black Prince), whose fiction I’ve always found contrived and artificial. The chronological imperative behind the selection occasionally became brutal. There was little time for second thoughts. One casualty of this process whom I deeply regret omitting is the Australian novelist Christina Stead’s The Man Who Loved Children (1940), a profoundly moving study of family life so pitch perfect that it’s hard to believe her novel is not better known.

Facebook Twitter Pinterest James Joyce, author of Ulysses (No 46), in 1934. Photograph: Lipnitzki/Roger Viollet/Getty Images

The vagaries of literary history never cease to astonish. Few in 1890, a year before Herman Melville’s death, would have placed any money on the future supremacy of Moby-Dick, as America’s greatest novel. In Britain, the smart money would have been placed on George Meredith, author of The Ordeal of Richard Feverel.

George who? In this game of “Humiliation” reputations continue to rise and fall on an invisible stock exchange of taste. In my lifetime, I have seen trading in DM Thomas (acclaimed author of The White Hotel, 1981) or in Bruce Chatwin (ditto, The Songlines, 1986), go from boom to bust. The living writers we most venerate are often the most vulnerable to long-term indifference and neglect.

Herman Melville’s Moby Dick, published 1851 (No 17).

There have also been some wonderful surprises. For years, creative writing at the University of East Anglia set the gold standard in the nurturing of new fiction. When, in 1992, the translated prose of WG Sebald burst on the scene with The Emigrants, it was an irony enjoyed by “Max” himself that he had already been working in comparative obscurity in the modern languages department of UEA for some 30 years. Today, Sebald’s reputation is more universally recognised than most, if not all, of the writers who have passed through UEA.

Meanwhile, although the novel in English retains deep roots in Britain and the United States, its expression as a multicultural phenomenon written in English, the world’s second language, is now taking new fiction into new territory. “In contemporary literature today,” Marina Warner remarked in a recent lecture, “we are seeing the written word cross linguistic boundaries” with an increasing interest in worldwide fiction. Warner suggested that literary translation may be part of a global trend of corporate homogenisation. This, she insisted, is not a problem, but a renewal: “Many forms of English are coming into being through these exchanges and crossings…”

My own prediction for the future of the novel is that, following every previous leap forward in literature through the ages, it is inconceivable that the changes in IT, communications, global capitalism and, above all, in human consciousness will not sponsor innovations to rival the modernist revolution of 1899 (Heart of Darkness) to 1925 (Mrs Dalloway).

Facebook Twitter Pinterest Virginia Woolf, author of Mrs Dalloway (No 50) circa 1933. Photograph: Central Press/Getty Images

Finally, we are left with the classics, often by dead white males, those books to which English language readers worldwide return again and again. Say what you like about my list (and thousands have merrily done so these past two years), the Anglo-American literary tradition, a source of some sublime and imperishable masterpieces, deserves to be celebrated for some astonishing achievements. Here, to provoke Observer readers just one last time, is my All Time Top 10 (chosen from this series, in chronological order):

Emma

Wuthering Heights

Moby-Dick

Middlemarch

The Adventures of Huckleberry Finn

Heart of Darkness

The Rainbow

Ulysses

Mrs Dalloway

The Great Gatsby

You pays yer money, and you takes yer choice.