In January 1963, two middle-aged Englishmen took tea together in the Christian quarter of Beirut. An eavesdropper might have mistaken their references to Eton, the Observer, and a nasty flu bug for genteel chit-chat. In fact, this exquisitely English encounter was the terrible end to a 30-year friendship corrupted and finally broken by the private and public betrayals of two lives devoted to espionage.

The transcript of this rendezvous is Ben Macintyre's scoop: the motor of an unputdownable postwar thriller whose every incredible detail is fact not fiction. In dramatic terms, what Macintyre calls "one of the most important conversations in the history of the cold war", bugged by MI5, marked the moment when one charming spy, Nicholas Elliott, began finally to extract a confession from another, his old friend Kim Philby.

For more than 50 years, the career of Harold Adrian Russell Philby – "Kim" to friends and family – has been the dark mirror in which we have read the bleakest episodes of Britain's postwar history. As Macintyre wisely acknowledges, Philby has inspired a voluminous bibliography that sometimes approaches literature, for instance in the novels of John le Carré.

Philby, codenamed "Sonny" by the Soviets, was the archetype of treachery. In 1951, his two co-agents, Guy Burgess and Donald Maclean, rumbled by the Americans, fled to the USSR. Philby's position seemed hopelessly compromised, but the establishment exonerated him. In 1955 the foreign secretary Harold Macmillan told the Commons that he had "no reason to conclude that Mr Philby has betrayed the interests of his country, or to identify him with the so-called 'third man', if indeed there was one." Soon after, "Sonny" landed a job as the Observer's Middle East correspondent. Once in Beirut, he resumed his career as a Soviet spy.

When he finally defected in 1963 – the climax to Macintyre's spellbinding narrative – the aftershocks of his ruthless treachery sent shudders through the establishment of which he had been so seamlessly a part. At first, he was – in Phillip Knightley's phrase – "the spy who betrayed a generation". Now, with most of that generation dead or in retirement, the Philby story has begun to morph into myth, slipping from history into psychodrama.

Enter Ben Macintyre, the author of Agent Zigzag and Operation Mincemeat, bestselling accounts of high-octane, covert capers from wartime Britain. Macintyre, the witty elegist of class-conscious, late-empire Britain, is the supreme pontiff of the "What larks" school of popular history. His challenge here, moving to the shady side of the post-imperial slope, is to unfold a harrowing tale while simultaneously entertaining his devoted readers with his customary galère of twits, alcoholics, transvestites, and fantasists. A Spy Among Friends, a classic spookfest, is also a brilliant reconciliation of history and entertainment.

At the heart of the Philby story, as Macintyre reports it, is a unique, and tragic, friendship between two public schoolboys, both spies. The children of cold and unreachable fathers, Elliott and Philby found in the secret world a comforting fraternity in which they could refashion their damaged selves.

Elliott, the younger man, became an MI6 high-flyer; Philby a KGB double agent. Bizarrely, each was spoken of as a possible "C", or head of the secret service. In keeping with the code of the club, they anaesthetised their lives with elaborate jokes, pink gin, and a passion for cricket. "They spoke the same language," Elliott's son told Macintyre. "Kim was as close a friend as my father ever had."

This relationship scaled Olympic heights of denial. Elliott said of his friend that "he did not strike me as a political animal". Such skewed intuition would come as no surprise to John le Carré, whose enthralling Afterword is one of the special pleasures of A Spy Among Friends. He remembers Elliott, who interviewed him for a job in the secret state, as a character from PG Wodehouse, "a quiet smile on his face, and the elbow cocked for the martini glass". Philby would not have had to work too hard to conceal his secret life from this Old Etonian bloodhound.

Astonishingly, Elliott never discovered that Philby had married an Austrian communist spy, or that his friend was a "penetration agent" for the KGB. Most dreadful of all, at the human level, he never knew that everything he confided to Philby in the bars of London's clubland, went straight back to Sonny's spymasters in Moscow. Part of the archetypal grip this story holds for the reader is as a case study in the existential truth that, in human relations, the Other is never really knowable. For both, the mask became indistinguishable from reality.

There are many deeper Philby-Elliott mysteries, judiciously anatomised by Macintyre. Why, for instance, did Philby resume his career as a spy once he had been set up in Beirut as the Observer's Middle East correspondent? The answer goes to the marrow of this astounding story.

For Philby, as much as Elliott, the secret life had become an addiction, a game of Russian roulette. With his friend Graham Greene, he could never wean himself from the thrill of risk, and like Macbeth became "in blood stepp'd in so far that, should I wade no more, Returning were as tedious as go o'er".

Back home, the mole-hunters nailed Philby's lies. Finally, the evidence was incontrovertible. In a cruel twist, MI5 decided that only one man could confront Philby. His best friend was sent to Beirut to extract the all-important confession. "I rather thought it would be you," said Philby, when Elliott turned up.

"If I had to choose between betraying my country and betraying my friend," declared EM Forster, in a famous credo, "I hope I should have the guts to betray my country." Elliott's interrogation led inexorably to Philby's precipitate defection. In that high noon among the teacups, each sacrificed everything: friendship, country and future. A Spy Among Friends is not just an elegy, it is an unforgettable requiem.