‘The cradle rocks above an abyss, and common sense tells us that our existence is but a brief crack of light between two eternities of darkness.” Thus Vladimir Nabokov memorably opens the first chapter of his autobiography, Speak, Memory (1966). That “brief crack of light” is what concerns us all, and the art form best constructed to examine and elucidate its many complexities is, I would argue, the novel. DH Lawrence defined it as the “one bright book of life” (once again luminosity is the metaphor).

Of course, all art is an attempt to come to terms with and explain the human condition. However, the way the novel has evolved makes it supremely successful in investigating our lives – or, more significantly, revealing other lives to us. You can write a long and complicated novel about a single day in an individual’s existence (Ulysses, Mrs Dalloway) or you can write a novel about war and conquest and the decline and fall of empires (War and Peace and the Fortunes of War and I, Claudius novels), but the novel’s unique power lies in its scrutiny of the human factor. No other art form – though theatre runs it close – can deal so effortlessly with the minutiae of our everyday lives. Crucially, no other art form can penetrate the subconscious mind so easily, can expose and elucidate the tiny shifts in nuance of a person’s behaviour and thinking. Other people are opaque, mysterious – even those closest to you. If you want to know what makes human beings tick, in every sense, good and bad, banal and sinister, read a novel.

And yet, looking at the 400-year history of the form there is, it seems to me, something of a lacuna. Novels have covered every possible aspect of human experience but very few have attempted to do full justice to the entirety of that “brief crack of light” that Nabokov describes. By this I mean the whole-life or cradle-to-grave novel. The genre is small and its exemplars rare. Even long novel sequences such as Proust’s A la Recherche du Temps Perdu or Anthony Powell’s A Dance to the Music of Time end long before their protagonist’s demise. The life described is not complete.

Steve Coogan in Michael Winterbottom’s A Cock and Bull Story, an adaptation of Laurence Sterne’s The Life and Opinions of Tristram Shandy. Photograph: Allstar/BBC

To attempt a definition: the whole-life novel must begin in early childhood or youth and follow, chronologically, the individual life through to that character’s death or near-death. One imagines that it’s quite common – Dickens, Thackeray, Trollope, Austen, Eliot, not to mention Balzac, Zola, Tolstoy, Melville, Twain et al … surely they wrote whole-life novels? But the great 19th-century novels, for all their compendiousness and range, always offered only parts of a life, however richly detailed these were. Taking on the long baggage of an individual’s entire existence is not only unusual in fiction, it’s also demanding, technically speaking: decades of living have to be compacted into a few hundred pages. The narrative sleight of hand required over necessary matters of elision, compression and summary is complex.

So what are the great exemplars of the whole-life novel? A varied collection of titles comes to mind. Perhaps the earliest is Goethe’s The Sorrows of Young Werther (1774). Another is Thomas Hardy’s Jude the Obscure. Both end with the death of their protagonists. Martin Amis has written a whole-life novel in reverse in Time’s Arrow – a grave-to-cradle novel. Hilary Mantel’s trilogy – Wolf Hall and Bring up the Bodies being volumes one and two – will eventually be an authentic and epic whole-life novel about Thomas Cromwell. Is Laurence Sterne’s The Life and Opinions of Tristram Shandy a whole-life novel, given that Tristram is not even born by the novel’s end, yet we know his destiny? Is Virginia Woolf’s abidingly strange novel The Waves an attempt to schematically cover the cradle-to-grave aspect of the human condition through her six characters’ monologues? Think of HG Wells’s The History of Mr Polly, Compton Mackenzie’s Sinister Street and its sequels, Giuseppe di Lampedusa’s The Leopard. Sometimes I wonder whether – at least in the Christian western world – the whole-life narrative that lurks behind all others, the Ur-narrative, as it were, is that of Jesus Christ in the New Testament. Christ’s birth-to-death story (by many hands) is relayed in tremendous detail and occupies the Nabokovian prenatal darkness as well as the posthumous one – which makes it unique. The eclecticism of this random list underlines the genre’s idiosyncrasy.

Olivia Cooke as Becky Sharp in the ITV version of Vanity Fair. Photograph: ITV

I first fell into the whole-life genre with my 1987 novel The New Confessions. I know its germination absolutely: I was trying to write a 20th-century version of Jean-Jacques Rousseau’s Les Confessions, published in 1782, one of the greatest autobiographies ever written. But Rousseau ends the account of his life when he was 53. I decided, for some reason, to take the narrative on into the central character’s very old age. At the end of the novel John James Todd – my Rousseau – is an old man. If not in his final senescence, he is waiting resignedly for his mortal clock to stop. His entire life is detailed in 500 pages.

As we read we can construct the parallel novel of our own life or postulate and prefigure how such a story might unspool

The whole-life novel is also a clear act of colonisation – by fiction – of what we now call life-writing: biography, autobiography and the intimate journal. But in fiction there is a different kind of authority. It is all, paradoxically, true – because it’s been invented by the author. Other forms of life-writing are highly suspect and partial. Donald Rayfield, Chekhov’s great biographer, has defined biography as “a fiction that has to fit the documented facts”. Beyond the documented facts the biographer cannot go or can only tentatively speculate. Not so in the whole-life novel. The fundamental “truth” of a life in fiction is guaranteed – this is it, this is what happened, this is what this person felt – and these novels exploit and relish that trust from the reader absolutely.

Having, somewhat to my surprise, now written five such novels – The New Confessions, Any Human Heart, Sweet Caress, Nat Tate: An American Artist and my latest, Love Is Blind – I am in a singular position to testify that the response from readers is often quite dissimilar to that afforded an orthodox novel. The reader comes to identify with the central character of the whole-life story that is being related in a different, more engaged way because they have all the necessary information, significant and insignificant. Parents, schooldays, love affairs, marriages, children, illnesses, bereavements, misfortune and good fortune, triumphs and disasters – all the biographical matter that slowly but surely fleshes out the adventitious elements of an individual life makes that life seem more real, somehow, less of a fictional construct. It’s the haphazard rollercoaster of a life that is key in the whole-life novel rather than a particular plot, theme or central relationship. Everything is grist to the mill for such novels and this is what distinguishes the type.

In the vast majority of orthodox novels, however, we are often ignorant of characters’ early lives or their middle age and old age or have only the sketchiest information. Even after the novel’s closure and catharsis the narrative of the rest of the life is left hanging, more often than not. We don’t really know what will become of Becky Sharp or Molly Bloom or – even – Harry Potter (though there is The Cursed Child and JK Rowling may steadily construct the whole of his life, I realise, in various media). However, there is one narrative with which we are intensely, intimately familiar – our own. I think that one of the greatest appeals of the whole-life novel is that we can see in a fictional alter ego’s journey from the cradle to the grave a paradigm or model of our own journey in all its aleatory and fascinating nature. As we read we can construct, if you like, the parallel novel of our own complete existence or, if we’re young, at least postulate and prefigure how such a story might unspool and be recorded. It gives the total life novel a powerful extra-literary frisson. It can be very beguiling for both the writer and the reader.

• William Boyd’s Love Is Blind is published by Viking.