Oleg Gordievsky was the most significant British agent of the cold war. For 11 years, he spied for MI6. That he managed to deceive his KGB colleagues during this time was remarkable. Even more astounding was that in summer 1985 – after Gordievsky was hastily recalled from London to Moscow by his suspicious bosses – British intelligence officers helped him to escape. It was the only time that the spooks managed to exfiltrate a penetration agent from the USSR, outwitting their Russian adversaries. It went some way towards exorcising the Cambridge spies, who a generation earlier had travelled in the opposite direction.

Gordievsky has told the story of his own improbable survival in a gripping 1995 memoir, Next Stop Execution. It charts his recruitment by the KGB, where his older brother Vasili served as a deep-cover “illegal”, and Gordievsky’s growing disillusionment with the grey totalitarian world of 1960s Moscow. There were stages in his journey. At an early age he learned German. He began reading western newspapers. Then as a KGB trainee he spent six months in East Berlin. He arrived just as the Berlin Wall went up, and woke one morning to the sound of tanks rumbling past the Soviet embassy.

But it was the 1968 Soviet invasion of Czechoslovakia that propelled Gordievsky towards the west and, as he put it, determined “the course of my own life”. By this point Gordievsky was a junior spy abroad, working for the KGB’s first directorate, and living in Copenhagen. He resolved to fight the communist system from the inside. His first tentative step was to call his then wife Yelena from an embassy phone bugged by the Danes, and to declare: “They’ve done it! I just don’t know what to do.” He expected an approach from western intelligence. It didn’t materialise.

Ben Macintyre’s wonderful The Spy and the Traitor complements and enhances Gordievsky’s first-person account. It reveals the dramatic role played by MI6 in recruiting and cultivating a serving KGB insider – and keeping him alive against the odds. Gordievsky’s British contacts were a colourful bunch. Some were upper-class cold war adventurers. Others were gifted working-class linguists recruited from Oxbridge. Women played a crucial part. All realised Gordievsky was unique.

Macintrye had no access to MI6’s archives, which remain secret. But he has interviewed all of the former officers involved in the case, who tell their stories for the first time. He spoke extensively to Gordievsky, who is now 79 and living in the home counties – a remarkable figure, “proud, shrewd and irascible”. The result is a dazzling non-fiction thriller and an intimate portrait of high-stakes espionage.

May Day parade in Moscow, 1970, the year in which Gordievsky was singled out by MI6. Photograph: Rolls Press/Popperfoto/Getty Images

Gordievsky’s double life started after a junior MI6 officer saw his name while leafing through a personnel file. It was 1970. A Czechoslovak spy, Standa Kaplan, had defected to Canada. Asked if he knew anybody who might be of interest to western intelligence, Kaplan threw out a few names, including that of Gordievsky, who was a friend from the KGB academy, aware of the drawbacks of communism, and not so different from him, he said.

In 1972 Gordievsky went back to Denmark for a second tour. MI6 was waiting. One morning its local head of station approached him while he was playing badminton at a suburban sports club. Lunch followed. Gordievsky’s manner during these early encounters was oddly calm, leading the Brits to wonder if they were the victims of a classic KGB “dangle”. Actually, Gordievsky had already decided to switch sides – a betrayal Macintyre calls “whole-souled” and “righteous”.

Gordievsky would go on to meet his British handlers once a month; he didn’t want money and said he was spying out of ideological conviction. Cassettes of these conversations were sent back to London in a diplomatic bag. Gordievsky, MI6 discovered, was a star asset. He had a prodigious memory and thorough knowledge of current and former KGB operations. At lunchtimes he would slip out of the embassy and hand over microfilm strips to his case officer for copying. These were Moscow’s secret instructions.

After three years, Gordievsky had to go back to Moscow. It was at this tricky point that MI6 came up with a plan to smuggle him out of the USSR, should the need arise. Its author was “Veronica Price” – a senior intelligence officer, working out of Century House, MI6’s unattractive HQ. The plan was called PIMLICO. London decided not to make contact with Gordievsky once he was home – too risky – but to have a procedure in place if he raised the alarm.

In Moscow Gordievsky survived a KGB interrogation, despite being drugged. He alerted MI6 and gave his minders the slip

The plan was simple, and almost comic. At 7.30pm on Tuesdays British officers would watch a bread shop next to Kutuzovsky Prospekt, a diplomatic complex. In an emergency, Gordievsky would turn up wearing a grey cap and holding a plastic Safeway bag. The MI6 officer would walk past him munching a Mars bar or a KitKat. This signal would trigger a plan to smuggle Gordievsky into Finland in the boot of a diplomatic car. A refresher memo was concealed in an OUP edition of Shakespeare’s sonnets.

After a long period of no contact MI6 was delighted when Moscow sent Gordievsky to London. In the meantime, he had learned English and closely studied the KGB’s British files. The book’s most contentious section concerns Michael Foot whom, Macintyre writes, the KGB cultivated for two decades. According to Gordievsky the KGB made regular payments to Foot in the 1950s and 60s, up until the Prague Spring. The cash went to fund the magazine Tribune. Macintyre writes that Foot wasn’t a Soviet spy. However, he does view him as “stunningly naive”.

In the summer of 1982 Gordievsky moved to London and began regular meetings at a Bayswater safe house with his MI6 handlers. For an extraordinary period he was briefing British and Soviet spies – and Margaret Thatcher, who referred to him as “Mr Collins”. MI6 boosted Gordievsky’s career by feeding him real, low-grade intelligence and by removing rival spies who threatened to expose him. There were close shaves, including an approach to the Soviet embassy by Michael Bettaney, a renegade MI5 loner. Betrayal eventually came from a venal CIA officer, Aldrich Ames, who tipped off Moscow – Ames is the traitor of the book’s title.

Macintyre touches only briefly on the unprecedented “download” of information given by Gordievsky to the west. It included details of the KGB’s attempts to influence western elections through “active measures”. In 1985 the KGB circulated a top secret “personality questionnaire”. It set out the characteristics it was looking for in a potential agent: narcissism, vanity, greed and marital infidelity. Soon afterwards, the Soviet government invited a prominent American, Donald Trump, to visit Moscow.

Thirty years on, Gordievsky’s successful escape still seems incredible. Summoned back to Moscow, he survived a KGB interrogation, despite being drugged. He raised the alarm with his supermarket bag and gave his minders the slip. Two cars driven by MI6 officers and their wives got him across the border. It was a hair-raising journey featuring a KGB pursuit, a smelly nappy and a bag of crisps. Gordievsky was forced to leave his second wife Leila and their two children behind in the USSR. They were reunited six years later but the marriage didn’t survive.

In later life, Gordievsky has been cantankerous and reproving. His accusations over the years have proved controversial and his claims have been interpreted very differently by the right and left in Britain. But there is no doubt that he played a profound role in undermining the Soviet system, not least by explaining the paranoia and fantastical thinking inside the KGB to western policymakers and agencies. His two books on Soviet intelligence operations with historian Christopher Andrew are invaluable. Gordievsky remains under sentence of death. He knows all too well the KGB’s view of treachery, and that his enemies play a long game.

• Luke Harding’s Collusion: How Russia Helped Trump Win the White House is published by Guardian Faber. The Spy and the Traitor by Ben Macintyre (Penguin, £25). To order a copy for £21.50 go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min. p&p of £1.99.