In 1938 W H Auden wrote a poem entitled “The Novelist”, in which he contrasts the figure of the Poet, “encased in talent like a uniform”, who “can amaze us like a thunderstorm”, with the Novelist who, alternatively, must “struggle out of his boyish gift and learn/How to be plain and awkward . . . among the Just/Be just” and –

. . . among the Filthy filthy too,

And in his own weak person, if he can,

Must suffer dully all the wrongs of Man.

The comparison between these two types of writing – an essential literary dichotomy – is telling and, even though Auden was thinking of his friend Christopher Isherwood when he wrote this, it seems to me a very acute general depiction of what the realistic novelist must achieve if he or she is truly going to excel. The contrapuntal tension implicit in the poem is very present in the novels of John le Carré, as this fascinating biography makes clear – and indeed can be seen as one among several marked dualities that characterise both the life of the man and the work itself.

Almost all biographies start with a summation of the subject’s antecedents. The first remarkable aspect of Adam Sisman’s compendious and compelling account of the life of David Cornwell, the man behind the le Carré pseudonym, is that by page 40 Sisman is still elucidating the career and character of Cornwell’s extraordinary father, Ronnie. Ronnie Cornwell (1906-75) emerges from this account as a near-mythic figure – a part-time millionaire and international jailbird, a fraudster and charmer, a shameless conman and stalwart Freemason, a moral blackmailer and proud father, a chronic philanderer and indefatigable optimist. Not surprisingly, having a monstrous, domineering father like Ronnie proved a complicated psychological impediment to the young David (who was swiftly enlisted by his father as gofer and bagman when required), a disadvantage compounded when Ronnie’s wife, Olive, left home for another man when Cornwell and his older brother, Tony, were respectively five and seven. Abandoned by his mother, left to the tender mercies of Ronnie and his various girlfriends, Cornwell has described his childhood as a time of constant embarrassment and trepidation.

But Ronnie wanted only the best for his sons, and they were expensively educated at prep and, later, public schools. David went to Sherborne – a place he loathed – and he left, of his own accord, early, in 1948, at 16, preferring to move to Bern, Switzerland, where he learned fluent German. It was at this stage of his education that he was first recruited by consular officials at the British embassy to report on local left-wingers and fellow-travellers and another duality entered his life, that of the spy.

Cornwell was an occasional recruit, run as an agent in the field, and this tenuous connection with the security services continued as he followed the usual pattern of the time – National Service (he became a junior officer, of course) and then to Oxford to read modern languages. While at university he undertook more covert work for MI5, spying on his left-wing friends and associates, searching their rooms, writing reports on their political proclivities. It was a betrayal that clearly troubled him – leading a double life exacts a price on your conscience. There was another price to be paid, too. Thus far, thus seemingly respectable – Sherborne and Lincoln College – but the seedy, roller-coaster world of Ronnie Cornwell continually dogged him.

Ronnie sometimes seemed vastly rich (there was a Bentley with the number plate RC1) but then the cash would run out or the lawsuits arrive. School and university fees were not paid and financial rescue had to be negotiated by kindly benefactors. David Cornwell, a young English gent in the making, was often only one step ahead of the bailiffs and, sometimes, the law. Ronnie used his son regularly in his dubious negotiations and David, at the age of 20 – this was the grim, diminished world of postwar Britain, remember – also lived the high-life that Ronnie’s deals provided: skiing in St Moritz, grand hotels in Paris and gambling holidays in Monte Carlo. There was the perfect persona presented to the world – David married young, a family was started, and after Oxford he became a master at Eton – and there was the flashy, occasionally sordid backstory (Ronnie had a violent streak and would beat his new wife as well as fleece his friends and neighbours). The complexities of being David Cornwell were multiplying.

And then he became a proper spook. In 1958 Cornwell joined MI5 in an official capacity and soon moved to MI6, where in 1961 his fluency in German secured him a posting as a second secretary to Bonn, his cover. Sisman tells us more about Cornwell’s life in espionage than anyone else has managed to achieve but even he admits to a few gaps. In any event Cornwell’s life as a spy was fairly short-lived, a matter of approximately three years. He had started writing novels and another double had emerged – “John le Carré”. The pseudonym was necessary because of his secret work: his early novels had to be submitted to a Foreign Office committee to be vetted for any potential leaks.

By now, 1963, he was the father of two sons and chafing under the pettifogging social niceties of being a junior diplomat. One gets the sense of a man who never felt at home in the classic English institutions that seemed to shape him: boarding school boy, Oxford undergraduate, Eton schoolmaster, army officer, Foreign Office diplomat. The dark, omnipresent buried reality of being the son of Ronnie Cornwell appeared permanently destabilising.

Escape arrived in the shape of his third novel, The Spy Who Came in from the Cold (1963). Cornwell was paid an advance of £175 by his publisher, Victor Gollancz. Within a year of publication he was a millionaire. Everything had changed. The book was a success of massive proportions – not just in Britain but also in the United States and France and Germany. Very swiftly John le Carré became a successful international author, his novels selling in the hundreds of thousands, with all the financial rewards that arrive with this unprecedented level of success. In fact, as Sisman establishes, although the publishers’ advances were subsequently huge, the novels that followed – The Looking Glass War (1965), A Small Town in Germany (1968) – didn’t generate the same deluge of money as his first book. Le Carré, as we should now call him, entered a world of movie deals and high-profile, high-spending living. However, his marriage was in difficulties and he was contriving to spend as much time as possible away from his wife, Ann, and their young family as they settled into provisional tax exile.

Le Carré became friendly with another high-flying young author, James Kennaway, a Scot, who had known similar sudden acclaim and gain from his debut novel, Tunes of Glory (1956). Kennaway – slightly older, raffishly good-looking, middle-class, a serial adulterer – was also a highly paid Hollywood screenwriter. They became very close friends, almost like brothers. All seemed really rather wonderful and heady as they shared the high-life together, planned movie scripts, speculated on future collaborations – and then le Carré began an affair with Kennaway’s wife, Susan.

Young novelist has affair – big deal. Except that this particular episode has produced three books, one from each of the participants in the love triangle: le Carré’s novel The Naive and Sentimental Lover, James Kennaway’s Some Gorgeous Accident and Susan Kennaway’s non-fiction account The Kennaway Papers. One of the unsought benefits of Sisman’s biography will be to quash the swirling rumours that have congregated around this liaison.

I myself was told, on good authority, that the affair between le Carré and Susan was deliberately engineered by Kennaway himself because he had writer’s block and needed a subject. The reality is more mundane – mutual attraction – but the consequences no less traumatic. The friendship between the authors was irretrievably damaged, although the Kennaway marriage survived until James died shortly afterwards (in 1968; he was 40 years old) of a massive heart attack. Le Carré and Susan remained in contact for a while. The affair has further literary interest in that The Naive and Sentimental Lover is perhaps the strangest novel (some would say the worst) in the le Carré canon and introduces yet another dichotomy that has a bearing on his life. The title is taken from concepts evolved by the German playwright Friedrich Schiller (le Carré’s work is full of references to German literature). Schiller differentiates between the naive man and the sentimental man – but the terms are not used in the sense that we commonly employ them. The naive man is spontaneous, natural, an artist. The sentimental man is “corrupted” by life in all its complexities and textures. One is reminded of Auden’s Poet figure, and his essential difference from the Novelist.

Le Carré saw himself as “sentimental” in the Schillerian sense and yet he longed to be “naive”, a true artist, as he thought Kennaway was. It is not too far-fetched to see these opposing themes enacted time and again in his life and his work.

Le Carré divorced his first wife, married again and had another son. He moved his principal residence to Land’s End while always keeping a base in London. After the success of The Spy Who Came in from the Cold, regardless of the critical fortunes of his subsequent novels, le Carré never really knew any kind of professional failure. He became an exceptionally rich man, published globally, rewarded, honoured, his novels routinely filmed.

In 1993, for example, he would receive for his novel The Night Manager an advance of $5m from his American publishers and more than £500,000 from his British. After the critical pasting he took for The Naive and Sentimental Lover, acclaim returned with the “Quest for Karla” trilogy – Tinker Tailor Soldier Spy and Smiley’s People – all adapted into legendary television productions. He has continued to be prolific, publishing a new novel every two or three years – engagé, disputacious, opinionated, still hard at work, now well into his eighties.

Inevitably the second half of le Carré’s life, that of a highly successful author, lacks some of the fizz and vim of the first half (Ronnie Cornwell’s death in 1975 brought a moment of real catharsis and closure for his son), but Sisman is excellent on the nuts and bolts of writing and of being published: inspiration, composition, deal-making, money accrued, critical reaction and so forth. As a professional novelist, I found the progress of le Carré’s later career compelling and revealing – changing publishers, changing agents, literary spats (with Salman Rushdie and Graham Lord) and the rest – but I can imagine it might not hold quite the same allure for someone not fully engaged in the business of being an author.

It must be difficult to write the life of a man who is still very much with us, and in the public eye, no matter how much liberty the biographer has been given to tell the story, warts and all. Sisman – a very fine and astute biographer – has done an excellent, not to say exemplary, job under the circumstances. Only rarely is one aware of a veil of discretion being drawn, of names not being named, yet it is impossible to imagine this Life being bettered – though le Carré’s own memoir, to be published in 2016, may add some gloss.

In considering this biography, a comparison comes to mind: there is something almost absurdly Dickensian about le Carré’s early life. He was abandoned by his mother as an infant; trusted to a corrupt, rackety and wilful father who was frequently bankrupted and imprisoned (as Dickens’s father was); tormented by feelings of class insecurity but eventually found fame and glory as a published writer under a pseudonym; and, in not-so-serene but well-heeled old age, recognised as a great English man of letters. Even his later so-called polemical novels have a whiff of the outraged Dickensian apostrophe about them, addressing the reader and pointedly making them aware of the injustice at large in the world.

Le Carré once wrote: “I saw the Berlin Wall go up when I was thirty and I saw it come down when I was sixty . . . I was chronicling my time . . . I lived the passion of my time.” So perhaps we should see him as our contemporary Dickens (there is also the mimicry, the voices, the relish of names), which, in the end, is not a bad epitaph: the Dickens of the Cold War. A designation that is all the more valid because it recognises his great merits as a serious writer.

What remains contentious about le Carre’s career as a novelist – and one thing he is very aware of – is another duality, namely that of the “genre writer” v the writer of serious literary fiction. The implication is that you cannot be one and the other at the same time. But what le Carré has achieved is to give the lie to this perceived duality in so far as it applies to spy fiction, for want of a better categorisation. Spy fiction, it seems to me, can’t easily be classified as a genre if only because so many so-called serious novelists have gladly and successfully taken it on. Joseph Conrad almost invented the category with The Secret Agent (1907) and Under Western Eyes (1911). Since then many novelists of the highest literary repute have written spy novels – Graham Greene, Muriel Spark and Norman Mailer, to name but three. In contemporary times so, too, have Ian McEwan, Sebastian Faulks and John Banville. In the same way that the fully achieved “historical” novel evades the genre category (no one would label The Spire, The French Lieutenant’s Woman or Wolf Hall as genre fiction) so, too, does the fully achieved spy novel.

The essential reason is easy to comprehend. Le Carré himself put it this way:

“I think all of us live partly in a clandestine situation . . . We hardly know ourselves – nine-tenths of ourselves are below the level of the water.”

The tropes of espionage – duplicity, betrayal, disguise, clandestinity, secret knowledge, the bluff, the double bluff, unknowingness, bafflement, shifting identity – are no more than the tropes of the life that every human being lives. The fully achieved, sophisticated espionage novel works precisely because in it you find all the troubling complexities of our own lives writ large. We all lie, we all pretend, we all betray – but in the spy novel you see those fundamental aspects of human behaviour, the human predicament, under a magnifying glass. The consequences may be more cataclysmic – walls may come down, bombs explode, deaths occur – but they find their exact and pertinent echo in our own quotidian experience.

What Joseph Conrad started, John le Carré enshrined and made modern. That is the real achievement of his great novels and why they will endure.

William Boyd’s most recent novel is Sweet Caress (Bloomsbury).