The celebrated author and former spy's popular books display a masterly understanding of moral complexity. His recent decision to switch publishing houses should see them firmly ensconced as modern classics

The announcement that John le Carré is moving from Hodder & Stoughton, his publisher for 38 years, to Penguin is not one that will unduly concern his legion of readers. It's what's published that matters, not who publishes it. Among the very few people who do take notice of imprints, though, are authors themselves.

For le Carré, who enjoyed his 78th birthday last month, the time has come to consider his legacy – there are 21 titles on the backlist – and where it will be best presented. With all respect to Hodder, it's not hard to see why a writer who is concerned with his place in history – and which writer isn't? – should want to relocate to the paperback home of modern classics.

Nevertheless, the question of where the author of such novels as Smiley's People and The Constant Gardener sits in English literature is much more than a matter of publishing houses. Since his first major success with The Spy Who Came in From the Cold, the 1963 novel that Graham Greene described as "the best spy story I ever read", le Carré has been seen as a great stylist, but the question remains of whether he is primarily a genre writer and whether that matters in terms of literary reputation.

He does not allow his novels to be entered for prizes and he's never formed part of a literary scene, but sometimes le Carré receives the highbrow approbation that his finest writing deserves. Philip Roth, for example, called A Perfect Spy "the best English novel since the war".

Le Carré recognises his "populist" appeal, but has said that his novels have been influenced by the German Romantic tradition: "They are for me a kind of Bildungsroman."

The model for the Bildungsroman is that of the innocent who, through a mixture of hard experience and good fortune, finds his way in the world. Although lonely boys and young men do populate le Carré's fiction, the creation for which he is best known is George Smiley, cunning, cerebral and middle aged, the very opposite of a naive youth.

Instead, Smiley embodied the grey shades with which le Carré recast cold war certainties. Le Carré, whose real name is David Cornwell, worked as a spy for British intelligence at the height of the cold war in the 1950s and early 1960s. He used the experience to create a fictional world, full of moral ambiguity, professional compromise and institutional corruption, that felt vividly real. The signature clarity of his prose was matched only by the distinctive murkiness of what it described.

The result, some critics suggested, was that he portrayed the Soviet bloc and the western allies as two sides of the same grubby coin. Last year, le Carré appeared to endorse this reading, telling an interviewer that, when he worked in espionage, he had considered defecting to the other side of the Iron Curtain. After the quote was published, he claimed that he had been misrepresented and that he had merely meant that he had made the imaginative effort to place himself "in the shoes" of the opposition.

What's beyond doubt is that le Carré had to rely on an inventive imagination from an early age. His mother left the family when he was five and he was told that she had died. He didn't see her again until he tracked her down when he was 21. His father, Ronnie, was a charming Dorset conman, a serial womaniser and a prison veteran who was determined that his two sons would be accepted into the English upper classes. "Respect, not money, was what he cared for more than anything," le Carré once wrote.

The family swung between great affluence and bankruptcy. The boys were often called upon to help their father evade creditors during an upbringing that le Carré has referred to as "clandestine survival". He and his brother, he has said, "were conspirators from quite an early age".

"People who have had very unhappy childhoods," he also observed, "are pretty good at inventing themselves. If nobody invents you for yourself, nothing is left but to invent yourself for others."

His troubled relationships with each of his parents proved instrumental in shaping his fiction. Duplicitous father figures crop up regularly in his work and, more obviously, the question of trust is at the centre of le Carré's fictional world. But there is also a strong current of recrimination running through his books that seems partly aimed at perfidious England, with all its carefully allotted privileges and discreet hypocrisies. And here, apparently, is the maternal source of his motivation.

"I think I am driven by some great sense of personal guilt," he once confessed. "I suppose if I were to stretch out forever on the couch, it would have something to do with feeling as a child that I had driven my mother away."

The young le Carré attended Sherborne public school, where he assumed the position that has since provided his main perspective on life, both as a writer and an individual: that of the outsider inside the heart of the British establishment. He disliked the school and left early to study languages at the University of Berne, before finally completing his education at Oxford.

In 1950, aged 18, he did his national service with the army's intelligence corps in Austria. Among his duties was interrogating escapees from the east. While at Oxford, he worked for MI5, joining far-left groups to look for Soviet spies. After leaving university, he spent two years as a master at Eton College, teaching French and German, and then became a full-time agent with MI5 in 1958. It was all invaluable preparation for the writing to come.

In 1960, he transferred to MI6, the foreign intelligence service. But two events brought his covert career to a premature end. The first was the exposure of Kim Philby as a Soviet spy, which meant that agents were compromised throughout Europe. And the second was the exposure of le Carré as the author of the bestselling The Spy Who Came in From the Cold. It was time to become a full-time novelist.

Le Carré has said that Philby was responsible for the deaths of countless British agents. He also acknowledged the widespread use of torture by the intelligence services in the Soviet Union. So it's a little strange that he should so often have suggested in his fiction a moral equivalence between east and west in the cold war. Since the collapse of communism, however, le Carré has sometimes seemed to have gone further than shared blame, suggesting the Americans are worse than anyone else.

He has in the past enjoyed other scraps of the political/literary sort. Le Carré argued against the publication of Rushdie's The Satanic Verses in paperback, pointing out that no one had the right to insult a great religion with impunity. Recently, in a half-hearted attempt at contrition, he acknowledged that his stance may have been wrong. "If so," he explained with characteristic sophistication, "I was wrong for the right reasons."

Tony Blair also became a hate figure, with le Carré declaring in 2005 that Britain was sliding towards being a fascist state. This new world view is neatly summed up by a character in A Most Wanted Man: "Americans are worse than you British, but they have an excuse… ignorance. They don't know what they're doing. But you English know very well."

Le Carré denies that his recent work has become more didactic. Rather, he has said, it amounts to a "clearer confusion" or "more articulate pessimism". Any tub-thumping might sit uncomfortably with the cultured, well-travelled and, by all accounts, charming companion who divides his time between houses in Cornwall and Hampstead.

Among his close friends are historian Timothy Garton Ash, Tom Stoppard and author William Shawcross; with Harold Pinter's death, he is now arguably the leading dissident in the world of letters. Le Carré does not approve of "artists getting medals from the state… I find it absolutely fatuous."

Like his early hero, Graham Greene, le Carré is at home in the company of diplomats and adventurers, at high tables and low dives. In his best, and most morally complex, work, he is acutely sensitive to thwarted idealism and human failing. He is married to Jane, with whom he has a son. His first marriage to Ann Sharp, which produced three children, did not long survive his change of profession in 1964. "I've had an untidy love life," he said a few years back, "and am now settled."

But romantic chaos can be a creative force, especially when it comes to envisioning believably flawed characters. Le Carré has shown a masterly understanding of the subtleties and psychology of deception and betrayal, what Garton Ash defined as "good men serving bad causes and bad men serving good". The difficulty comes with bad men serving bad causes – more frequent in recent novels – which might be more difficult to render successfully.

In any case, le Carré's legacy is almost certainly assured. In so evocatively capturing the art of spying, he is the writer who turned spy fiction into an art form.

The Le Carre lowdown

Born: James David Cornwell, 19 October 1931, in Poole, Dorset. Married twice. Has four children. Lives with his second wife in Cornwall.

Best of times: Perhaps leaving the Foreign Office in 1964 because he was then able to write full time; in the same year, he won the Somerset Maugham award. By his own reckoning, his best work includes The Spy Who Came in From the Cold, Tinker, Tailor, Soldier, Spy, The Tailor of Panama and The Constant Gardener.

Worst of times: Several contenders from his childhood and early adulthood – including his runaway mother, who abandoned the family when he was five, and his conman father.

What he says: "The monsters of our childhood do not fade away, neither are they ever wholly monstrous. But neither, in my experience, do we ever reach a plane of detachment regarding our parents, however wise and old we may become. To pretend otherwise is to cheat. "

"You can't make war against terror. Terror is a technique of battle. It's a tactic that has been employed since time immemorial. You can conduct clandestine action against terrorists and that must be done."

What others say: "When one thinks about [le Carré's] books, what comes out is a very patriotic man. I think his patriotism includes the whole concept of democracy and responsible political behaviour." Critic Al Alvarez