David Cornwell, or John le Carré, as we usually call him, must have been wincing as he viewed the CCTV stills of those two Russian visitors to Salisbury. He was surely shaking his head in dismay as he watched their later avowals of their touristic interest in Salisbury Cathedral on Russian TV. This is not just because he is an old secret service hand himself, with some professional appreciation of the skill at avoiding surveillance that any decent agent must develop. It is also because his every novel takes it as axiomatic that those with covert purposes (Russian spies, British spies, terrorists) have to be brilliantly cunning at hiding themselves. If spooks were to be like the apparent blunderers of the GRU, le Carré’s entire oeuvre would be doomed.

The target of all the machinations of his 1983 novel The Little Drummer Girl, now made into a glossy BBC six-part adaptation, is certainly as ingenious and elusive as any of his previous masterminds. He is Khalil, an accomplished and highly intelligent bomb-maker, dedicated to the murder of prominent Jews in western Europe at the end of the 1970s. The BBC’s previous le Carré adaptation, The Night Manager, was updated from its original setting in the early 1990s to something like the present. The protagonist’s scalding experiences undercover in Ireland were turned into traumatic memories of missions in Iraq. The Little Drummer Girl takes us back to 1979 and stays there.

We are in the world before the collapse of communism. The Little Drummer Girl was the novel that le Carré wrote immediately after Smiley’s People, in which his owlish British intelligence officer George Smiley had finally defeated his opposite number in Soviet intelligence, Karla. It seemed to mark a new direction. It focuses not on the cold war spy game between west and east, but instead on the consequences for western Europe of the conflict between Israelis and Palestinians. And it was the first, and remains the only, le Carré novel with a female protagonist. In this most traditionally masculine of genres (another former spook, Stella Rimington, has been one of the few to break the gender cartel), this felt like a significant experiment for the novelist.

The elaborate construction of a credible lie is the essential process of a le Carré novel

The fall of the Berlin Wall may have been one of the happiest events of John le Carré’s lifetime, but it was a cruel deprivation for him as a novelist. The wall itself runs like a beloved line of narrative through le Carré’s fiction for almost three decades. In 1963 he transformed the spy fiction genre with the publication of his third novel, The Spy Who Came in from the Cold, a novel that begins and ends at the Berlin Wall. Moral compromise – even moral failure – became le Carré’s stock in trade. The world of espionage he presented was a far cry from the glamour and gadgets of many of his contemporaries.

Just a few months before the novel appeared, Ian Fleming published On Her Majesty’s Secret Service, in which James Bond infiltrates the Alpine headquarters of Ernst Blofeld and defeats Spectre’s scheme to destroy British agriculture. A very different kind of secret service. Le Carré was not entirely alone in bringing some kind of “realism” to the genre. In the 1930s, the derring-do of Erskine Childers and John Buchan had given way to the more unsettling fiction of Eric Ambler, in which well-meaning amateur heroes – journalists or writers – find themselves caught up in the machinations of spies and international criminals. Yet the plots were often baroque, while the protagonist was usually appalled by the world of intrigue into which he had blundered.

Facebook Twitter Pinterest More than a touch of James Bond … Hugh Laurie and Tom Hiddleston in The Night Manager. Photograph: BBC

In 1962, Len Deighton published his debut novel, The Ipcress File, also a bestseller. The first of a series of successful spy novels, it has something in common with le Carré’s murky realism: it gave the reader cynical British spymasters and a secret service governed by turf wars and bureaucratic imperatives. But the narrative, involving the kidnapping and brainwashing of a series of British boffins, was more Bond than Smiley, and the novel’s multiple and sometimes highly contrived killings took it a long way from probability.

Le Carré brought a new authority and, the reader inferred, authenticity to the cold war thriller. It soon became widely known that “le Carré” was the pen name of David Cornwell, who had worked for several years for MI6, and was still doing so when he wrote The Spy Who Came in from the Cold. The book’s success allowed him to become a full-time writer, and for the next 25 years he produced a string of grimly plausible cold war novels. The last was The Russia House (1989), a novel about a thawing Soviet Union. He brought to spy fiction not just a special circumstantial texture – that language of “tradecraft” – but also a beauty and discipline of plotting that is rarely found in the fiction of his predecessors.

When his plotting is at its best, as in Tinker Tailor Soldier Spy (1974) or The Perfect Spy (1986), it is a rare literary achievement. These stories feel distant from the adventures of The Night Manager, especially when reimagined for television with more than a touch of James Bond. In Tinker Tailor, Smiley spends most of his time either reading through old files or interviewing old agents. The very plot of the novel is about the piecing together of narratives, of inconsistencies or echoes between different stories. It is slow and minutely attentive. No wonder that, for the aficionado, the 2011 film is so disappointing, at once reductive (all the stages of investigation hurried into just two hours) and visually extravagant (in order to distinguish itself from the meticulously faithful 1979 TV adaptation).

Facebook Twitter Pinterest Murky realism … Gary Oldman in Tinker Tailor Soldier Spy (2011). Photograph: Focus/Everett/REX Shutterstock

In The Little Drummer Girl, the shift back to a time without mobile phones or the internet – and with almost nothing in the way of video surveillance – emphasises the need for narrative and psychological ingenuity. The TV drama, like the novel, relies on operatives with binoculars and large cameras, gnomic calls from public telephones, coded messages scribbled on pieces of paper. It is gratifyingly cluttered with the antique machinery that spies might have loved four decades ago: reel-to-reel tape recorders, ticker-tape machines and primitive video recorders. The plot relies on the skills of forgers of documents and photographs.

Like The Night Manager, The Little Drummer Girl is a story of infiltration. Charlie (AKA Charmian), a young would-be actor, has been picked by Martin Kurtz, a wise and wily Mossad spymaster, to infiltrate bomb-maker Khalil’s tightly controlled network. Kurtz has chosen her because of her righteously radical past (even stronger in the book than the dramatisation); to the terrorists she will seem a likely recruit. He has also picked her because she is currently touring half-empty repertory theatres, and he can lure her with the offer of a real star performance. But will she be persuaded?

The elaborate construction of a credible lie is the essential process of a le Carré novel. How do you make the most suspicious person in the world believe your lie? Kurtz and Charlie’s handler “Joseph” instructs her in the arts of disarming or deflecting suspicion. Early le Carré novels are all masterclasses in these arts. Karla, the Soviet espionage chief in Tinker Tailor may be a communist fanatic, but he is a subtle analyst of the human heart. He instructs Bill Haydon to begin an affair with Smiley’s wife in order to discredit Smiley’s suspicions about him, not only with his superiors, but with himself. The scrupulously rational Smiley will naturally check himself from suspecting the man who has cuckolded him; he would only be giving in to resentment. In The Night Manager, the shrewd and calculating arms dealer Richard Roper lives by suspicion, yet is made willing to look past the coincidence of Jonathan Pine turning up more than once in his life by an elaborately planned MI6 back story, in which Pine is on the run from a murder that he has committed. He is compromised and therefore manipulable, and Roper cannot resist him.

The moral failure of western intelli­gence was essential to his stories. But since 1989, their spooks have begun looking like villains too

Spy fiction is narratively paranoid. Charlie becomes alive to the possibility that anyone could be a spy. Queuing to cross the border between Greece and Yugoslavia, she finds herself behind an open sports car, two men in the front, two women in the back. Are they Joseph’s? Or Khalil’s? Or police of some kind? “She was learning to see the world that way: everyone belongs to someone.” She begins to see how little the rest of us see, how unknowing are all “the innocents” around her. “They are where I came from, she thought. They are me before I walked through the looking-glass.” But now she is awake to possibilities. The world is charged with meaning. In a spy novel, everything that the initiated character notices might mean something. Suspicion electrifies narrative.

In 1983, someone reading The Little Drummer Girl must have wondered where the book’s sympathies lay – or rather, where the reader’s sympathies were being directed. Even more so now. Kurtz is a liberal Israeli who wants to make the conditions right for a peaceful solution in the Middle East. Dates matter very much. Kurtz, in his 50s in 1979, is a Holocaust survivor with memories of a winter journey in a cattle truck and of watching his own mother going to her death. In the TV version, an awkward west German secret service officer notices the tattooed number on Kurtz’s forearm. In a passage not to be found in the book, Kurtz jokes about how the numbers, tattooed when he was young, stretched out as he grew. The Israeli agents in west Germany shudder inwardly at the country in which they operate.

An author’s note at the opening of The Little Drummer Girl acknowledges the “many Palestinians and Israelis who gave me their help and time in the writing of this book”, ranging from a former chief of Israeli military intelligence to a Palestinian military commander. The even-handedness is decisive and is built into the plot. The Mossad agents instruct their recruits for undercover work using the best arguments of the terrorists. They have to understand how it feels to believe in their enemies’ cause.

Where is our judgment to settle? It is a question that the reader of le Carré’s post-1989 fiction rarely has to ask. Since 1990, many of the enemies le Carré has found for his troubled, itchy agents have been straight-up villains. In The Night Manager, Pine thinks of international arms salesman Roper as “the worst man in the world”. And so he is – a man driven to accumulate ever more wealth by trading the most effectively murderous weapons to anyone who can pay his prices. In one respect the TV dramatisation improved on the book: Hugh Laurie brought a ghastly charm to the character that was entirely absent from the novel.

Some of le Carré’s more recent villains have been corporate, and they too are unambiguously bad. In The Constant Gardener the agent of evil is a multi-national pharmaceutical company, responsible for the murder of diplomat Justin Quayle’s wife Tessa, who was about to reveal its deadly use of experimental drugs in east Africa. The novelist’s values and moral energy may be impeccable, but the reader can feel their pressure. The new heroes may be psychologically wounded – the sine qua non for a le Carré protagonist – but there are no doubts about what they are up against.

The moral failures of western intelligence agencies were always essential to the stories le Carré told, the cynicism of spymasters being a firm convention of the genre. But since 1989, the western spooks have begun looking like villains too. In The Constant Gardener, the Foreign Office sides with murderous big pharma. In Absolute Friends, the hero, Ted Mundy, is pitted against the “renegade hyperpower” that is the United States and eventually becomes awkward enough to be killed by his own side. It is notable that the struggle against Islamist fundamentalism, which has become the major preoccupation of secret services in western Europe, and is the background to many recent film and TV thrillers, has interested le Carré the novelist very little.

Facebook Twitter Pinterest Battling big pharma … Rachel Weisz and Ralph Fiennes in The Constant Gardener (2005) Photograph: Sportsphoto Ltd/Allstar

His most recent novel, A Legacy of Spies, is a revisiting of the cold war and, indeed, of the “case” of Alec Leamas in The Spy Who Came in from the Cold. It is narrated by that old Circus hand Peter Guillam, now retired but ready to take us back to that looking-glass war. Many readers detected a nostalgia for the days before the wall fell. The Berlin Wall was where we last saw Smiley, waiting to see if his final gambit against Karla would work in the last chapter of Smiley’s People: “the white halo of sodium arc lights, and behind it … a barricade, a pillbox, then the bridge”. Germany is at the heart of all le Carré’s early novels (Cornwell’s German language skills were important in his own secret service work) and in The Little Drummer Girl we return there.

We also return to the business of psychological manipulation that gave literary urgency to le Carré’s new species of spy fiction. In The Little Drummer Girl, the canny Kurtz patiently explains to his sceptical superiors how Charlie’s fiery leftist past will make her the best recruit. “Her ideological drift did not dismay him in the least; the nearer she was to drowning, he said, the greater would be her pleasure at coming aboard.”

Charlie’s sagacious Mossad advisers are experts in lying, as are the Circus veterans who advise Leamas how to deal with interrogation. Interrogation, the fascination and the slowness of it, is essential to The Little Drummer Girl. It is a long novel, dedicated as much to manipulating those who are questioned as to finding out any answers. The patient slowness of these sessions is what makes them credible.

When he is as careful and subtle as this, le Carré is not necessarily easy for TV. The Night Manager sometimes resembled a Bond story. All those glamorous settings! Where was the drabness of vintage le Carré? The hopeless Somerset prep school where we find Jim Prideaux at the beginning of Tinker Tailor; the bed and breakfast in a deserted Devon coastal resort where Magnus Pym is seeking refuge on “a blustery October morning” in the first chapter of A Perfect Spy. These are the places that a British spy ends up. (You can still see the influence on contemporary thrillers: the British spies in Phoebe Waller-Bridge’s Killing Eve – which is based on Luke Jennings’s Codename Villanelle novellas – live in the drab old world, while the female assassin has a lovely Paris flat and a limitless expense account.)

Tinker, tailor, writer, spy: the many lives of John le Carré, in his own words Read more

You become a spy because of something in your past. The idea is alive in The Little Drummer Girl too. Charlie has been driven by her own unhappiness into rebellion and excess, and, before the spooks ever get to her, has already constructed a fiction about her past to justify her bohemian life. She is entirely ready to adopt a new set of lies. The novel emphasises her restless, almost angry, promiscuity; the TV version, written in a different age, avoids this. (Detectably, the sexual politics of the novel has given the writer and director of the adaptation some uneasiness.)

A Perfect Spy, le Carré’s most autobiographical novel, draws on this interest in the early habit of duplicity. It interleaves its account of Pym’s career in the British secret service with his memories of his relationship with his father, Rick, a charming conman and dedicated deceiver. The sense that the arts of deception are first learned within the family is made all the stronger by our knowledge that the character of Rick Pym is based closely on Cornwell’s own father, Ronnie. Pym was born in the same year as le Carré and has had the same public school and Oxford education. The spy-turned‑novelist learned early and painfully about secretiveness – and made it his gift to the British novel.

• The Little Drummer Girl starts on BBC One on 28 October.