Among the many questions that surround the Cambridge spies, one has occupied historians ever since the scale of their treachery became fully known. Why did they choose to betray their country? Several reason are given why Guy Burgess, Kim Philby, Donald Maclean, Anthony Blunt and John Cairncross – commonly known as the Cambridge Five, though there may have been others – decided to serve the Soviet state. In the 1930s they saw the USSR as the chief bulwark against the advance of Nazism and fascism; in the Second World War, they acted in response to Britain and the USSR being allies; during the cold war, they viewed the United States as the chief threat to world peace. Above all, the spies had an overriding ideological commitment to communism. Acting on this was more important for them than clinging to old loyalties of king and country.

No doubt all of these factors played a part, but they are less than thoroughly convincing. The spies were recruited in the 1930s, when the danger of Nazism was becoming clear; but they continued to serve the Soviet Union after it entered into a pact with Nazi Germany, when many other communist sympathisers fell away, and went on serving the Soviet state after it ceased to be Britain’s ally. As to the idea that they were devoted to an idea of communism, they remained loyal to the Soviet Union long after it had become clear to practically everyone – including many who had been communists – that it would never be a workers’ paradise.

A more compelling reason for what the spies did may be found in the words of Kim Philby, the only one to have written a book-length account of his career in espionage:

How, where and when I became a member of the Soviet intelligence service is a matter for myself and my comrades. I will only say that, when the proposition was made to me, I did not hesitate. One does not look twice at an offer of enrolment in an elite force.

These are the final words of Philby’s introduction to My Silent War, first published in 1968. There is nothing in them about ideological conviction. Instead he chooses to focus on the elite character of the organisation he joined when he made the decision to work for the Soviet state. In so doing, he gives away more than he intended of the motives that led him and his friend Guy Burgess to act as they did. They were convinced it was about to become the world’s dominant power. By switching their allegiance as they did, they could move from the British elite to which they belonged to the Soviet ruling class. But because the Soviet authorities never fully trusted any of their foreign agents, things didn’t work out that way. When Burgess arrived in the Soviet Union along with Donald Maclean after the two disappeared suddenly in 1951, he discovered that he was cut off from any meaningful connection to power. He found this situation almost intolerable.

Four years before he died in exile, Burgess told a Canadian freelance reporter: “My life ended when I left London.” He hadn’t left everything of his life behind, however. Another journalist who knew him in Moscow around this time remembered “seeing him at various parties in a grey suit, rather stained and baggy, and wearing an Old Etonian tie”. Some version of this tie was Burgess’s badge of identity all his adult life. He was wearing it when he turned up unshaven and hungover at his club in London, while taking part in working-class hunger marches, during his trips to America (which he despised), in his years in Russia, where he wore an OE bow tie together with the Order of the Red Banner that he claimed to have received for services to the Soviet state, and on his deathbed when he expired from too much drink in Moscow in 1963.

Burgess’s school tie seems to have meant more to him than anything else. But his time at Eton was not altogether successful. Good-looking and intelligent, he played football for his house, ran and rowed, joined the Eton College Officer Training Corps, read papers at an essay society, received art and drawing prizes and contributed drawings to college magazines. But the supreme prize of life at Eton – membership of Pop, a self-selected elite of between 24 and 28 boys who enjoyed privileges such as wearing ­coloured waistcoats and caning other boys – eluded him. One reason for his failure, Andrew Lownie suggests, may have been that this elite group tended “to prefer aristocrats rather than the sons of naval officers”.

A central theme of this exhaustively researched and absorbing book, the first full biographical study and likely to remain the definitive life, is that its subject was unknowable. “Burgess is certainly the most complex and enigmatic of the Cambridge spies,” Lownie writes, “a man of enormous contradictions and complexities.” It is true that Burgess was a different type of person from the four other men. Where the others were highly self-controlled and shunned public scrutiny, he was wildly flamboyant and often embroiled in scandal because of his drunken behaviour. But was he so inscrutable? Or was he just an extreme embodiment of his class and his time?

Born of Huguenot stock in Devon in 1911 into a military and naval family, Guy Francis de Moncy Burgess was educated at Eton and Cambridge, where he became a member of the Apostles, and went on to hold positions in the BBC, the intelligence services and the Foreign Office. Having joined a communist cell at Trinity College, Cambridge, he attempted to join the Communist Party of Great Britain but the party was wary of middle-class members and told him he would have to wait for five weeks. He had no difficulty in making contact in 1934 with a Soviet “illegal”, Arnold Deutsch, who had already recruited Philby. Deutsch told him to leave the CPGB and publicly renounce any sympathy for communism, which he did. By 1936 he was working for MI6. In an irony that was not lost on him, one of the tasks he was given was uncovering secret party members in Oxford and Cambridge.

Burgess’s assets as a spy were his network of friends and his charm. His company was sought out by Maynard Keynes, W H Auden, Winston Churchill, Isaiah Berlin, Harold Nicolson, George Orwell, Lucian Freud and many others. His homosexuality wasn’t particularly important in his decision to become a Soviet agent. Spies lead double lives, as gay men were forced to do in Burgess’s day; but few gay men became Soviet spies and most of the spies were not gay. He usually ascribed his sexuality to his years at Eton, though he told some people that a traumatic incident he suffered at the age of 13 when he heard a scream from his parents’ bedroom, where he found his mother trapped under the body of his father, who had died while making love to her, may also have played a part. Given that Burgess was a lifelong fabulist whose account of events could never be trusted, it is impossible to know whether this is true (although his father’s death, caused by heart disease, certainly left its mark on him).

In looking for reasons for Burgess’s decision to work for the Soviet Union, his sexuality is largely irrelevant, as indeed is his personality. As Lownie writes, “Burgess was the product of his generation.” There was nothing unusual in his attitude to communism or the Soviet Union. At the time, pretty well everyone who mattered in intellectual circles believed that liberal individualism was finished. Even many who saw little that was appealing in the Soviet system were convinced that it was winning. The Great Depression had discredited capitalism, and it seemed obvious that the future lay with a planned society.

But there was something more in the appeal of the Soviet Union for Burgess. Lownie identifies it: “Just as the 19th century had belonged to the British empire, Burgess felt the 20th century would belong to Russia.” Here, Philby’s career may be instructive. He wasn’t the only member of his family who betrayed his country. In the early 1920s his father, Harry St John Philby, also known as Sheikh Abdullah, a British Arabist and colonial intelligence officer, staged a conversion to Islam and became chief adviser to the Saudi king Ibn Saud. In this capacity, he advised the king that Hitler would win the Second World War, recommended the Sauds disinvest from Britain and brokered a deal with US oil interests. Though mistaken in predicting victory for Hitler, Philby Sr was prescient in his assessment that British imperial power was on its way out. His reward was wealth, and a 16-year-old Saudi slave girl whom he married. He died while visiting Kim in Lebanon in 1960 and his last reported words were: “God, I’m bored.”

Philby’s decision to spy for the Soviet Union becomes clearer once it is seen as being – like his father’s – a switch from a declining to a rising power. In the case of Burgess, there is no family history of playing the great game of empire, yet his attitude to the country to which he transferred his loyalties had a distinctly colonial ring: “I simply loathe Russia. I’m a communist, of course, but I’m a British communist, and I hate Russia!” Like quite a few western communists, he combined a profound reverence for the Soviet state with a deep contempt for Russian culture and Russians. He knew of the purges of the 1930s, the mass peasant die-off that accompanied agricultural collectivisation and what it meant to be consigned to the labour camps, but he was unmoved. When he complained about Soviet living conditions it was to protest against the quality of Soviet-made pyjamas, which he objected were unfit to sleep in. Probably the most lasting impression left on him of his time in the Soviet Union was a stainless steel tooth, inserted in the early years of his exile after a street thug attacked and knocked him down for his watch while he was living in a city on the Volga.

It may be hard to fathom Burgess’s inner world, but that doesn’t make his life unreadable. His scruffiness and drinking were excessive but hardly unconventional; a more moderate version of his behaviour was practically mandatory in the bohemian literary circles that he inhabited for much of his life. His opinions were the truisms of his era. If he went further than most in applying them, it may have been simply because he was more arrogant and careless. His fellow Etonian George Orwell showed much more individuality when, rather than going to Cambridge and joining the Communist Party like so many of his generation, he signed up to serve as a military policeman in Burma instead.

As an epigraph to the last chapter of Stalin’s Englishman, Lownie cites Alan Bennett, the scriptwriter for the TV drama An Englishman Abroad (1983), in which Burgess is shown as melancholy and homesick during his Moscow exile (much as he was, in real life). About this, Bennett, the authentic voice of bien-pensant respectability, wrote: “I find it hard to drum up any patriotic indignation over Burgess or Blunt, or even Philby. No one has ever shown that Burgess did much harm, except to make fools of people in high places.” The facts suggest otherwise. Between 1941 and 1945 he passed more than 4,600 documents to Moscow. He had access to secret information about the postwar peace conferences and the founding meetings of Nato, the UN and the OECD. In the 1950s he advised Soviet intelligence on recruitment and, according to a defector, helped plan a homosexual blackmail operation against a friend serving with a British diplomatic mission.

Loyal to neither his friends nor his country, Burgess was far from being a harmless joker. His career was an episode in the dissolution of Britain’s imperial elite; looking to join the coming ruling class, he ended up on the fringes of a system that collapsed not much more than a quarter-century after he died. His life was the stuff of comedy rather than tragedy. In 1957 he wrote to the then Conservative chancellor of the Exchequer, Peter Thorneycroft, addressing him as Peter and asking for income from a trust fund paid into his bank account with Lloyds on St James’s Street to be released to him in Russia, so that he could settle bills with Fortnum & Mason, his shirtmaker, his tailor and Collet’s the bookseller. The letter worked and the money was transferred. Two months later Burgess was classified as a non-resident British subject. He remained non-dom for the rest of his days, consumed by sickly nostalgia for the country whose privileges he cherished and betrayed.

John Gray is the NS’s lead book reviewer