Writing about Whittaker Chambers, who, from being a member of the Communist Party and later an ­operative for Soviet intelligence, defected to become a leading figure of the American right, Daniel Oppenheimer comments: “The party, for Chambers, was less a political organisation than a crucible for the forging of his own Bolshevik soul.” It is an astute judgement, one that applies to some degree to all of the six apostates from the left whose lives and beliefs Oppenheimer examines in this arresting and at times revelatory study. Chambers, James Burnham, Ronald Reagan, Norman Podhoretz, David Horowitz and Christopher Hitchens were different from one another in many ways, some of them quite fundamental. And yet, for each man, his volte-face was much more than a response to world events: it was an exercise in self-creation, in which what was being fashioned was the meaning of a life.

Chambers turned to communism while fleeing a fractured family and a hidden life of homosexual cruising, which he renounced when he rejected communism and became a Christian. A scion of Princeton and Balliol, Burnham used his mastery of Marxist theory to show off his formidable intellect and, once he had demolished Marxism, to issue dramatic forecasts of the near future that were nearly always confounded, but somehow enabled him to enjoy an afterlife as a consultant to the CIA. Reagan moved on from liberalism to the American presidency by way of a failed marriage and a ­flagging Hollywood career. Horowitz shed his leftist allegiances during a mid-life crisis that enabled him to rediscover his Jewish identity. Podhoretz became a liberal and then an ex-liberal in order to savour the sweet and sour taste of success. Hitchens melded the rakishly contrarian persona he had fashioned over the years with a new identity as a booster of American power in what he chose to see as a war of liberation in Iraq. For these apostates, politics was an intensely personal engagement. Though with some it involved a conversion from one faith to another, for all of them the journey to the right mixed the qualities of a ­pilgrimage with a continuing need to believe in themselves.

It’s worth reflecting on how the right has changed as a result of this migration. Ever since it emerged in the late 18th century as a distinct tradition of modern thought, conservatism has been defined by a suspicion of grand schemes of world improvement. Whether their thinking was grounded in a religious belief in original sin (as in the cases of Edmund Burke and the American conservative Russell Kirk) or a sceptical view of the power of human reason (as in David Hume and Michael Oakeshott), conservatives distrusted any attempt to remake the world according to the dictates of high-minded ideals and abstract models.

Old-style conservatism embodied a deep-seated mistrust of visionary politics and an unapologetic embrace of imperfection. In Britain, Margaret Thatcher set in motion a transformation in which this tradition was supplanted by neoliberal ideology, but the left had little input into the process. In the US, however, the migration of leading figures from the left played a critical part in converting the right to a type of militant progressivism. Having abandoned their socialist beliefs but not the evangelical mindset of the radical left, they demanded that democratic capitalism be exported across the globe, turning the right into a vehicle for a vision that was as ambitious and as delusional as those they had rejected.

Oppenheimer’s six pilgrims began their journeys from rather different points on the left. Whereas Chambers started off a loyal Communist, Burnham and Hitchens began as Trotskyites. During most of the time Horowitz was active on the left he thought of himself as a Marxist; but he never treated Marxism as an orthodoxy, and his principal involvement was with the Black Panthers. Reagan began as a Rooseveltian New Dealer and was at one point a member of the United World Federalists (which campaigned for world government), while Podhoretz, when he set out on his journey to neoconservative punditry in the early 1960s, was what Oppenheimer describes as “an officer in excellent standing of the left-liberal cultural establishment”. All six men viewed the society in which they had grown up as directionless, and found in their left-wing convictions a sense of mission.

When Chambers wrote that communism “offered me what nothing else in the dying world had power to offer at the same intensity – faith and a vision, something for which to live and something for which to die”, he spoke for many in Britain and America who joined the Communist Party in the interwar years. But, from the first, there was an unreality to these visions of a new world, together with an element of dark comedy. Chambers joined the Communist Party of the United States in 1925 after reading one of Lenin’s duller works, a pamphlet titled The Soviets at Work, published only months after the Bolshevik revolution and based on a speech dealing with organisational problems of Soviet life. For Chambers, Lenin’s words came as a revelation. “The reek of life was on it. This was not theory or statistics. This was socialism in practice.”

Yet if Chambers found life in Lenin’s pamphlet, it was a life of which he knew nothing. He never visited the Soviet Union. The communists he knew were those he worked with in the New York offices of the Daily Worker, who were mostly like himself: disoriented refugees from the middle classes who found relief from loneliness and lack of purpose in drinking and sleeping with one another, discussing the collective farms that would soon be established in America and the inspiring challenges of engineering the souls of the masses.

Chambers glimpsed working-class communists in Berlin – “little knots of furtive figures selling newspapers” – when he visited the city in the summer of 1923, and during encounters with working men when he was looking for casual sex. But in general, working people were notable by their absence from his life. With the exception of Reagan, who worked as a trade unionist in the Screen Actors Guild and later as a PR man for General Electric, the same is true of all of Oppenheimer’s subjects. None of them had any deep knowledge or understanding of the people to whom they avowedly dedicated their lives.

Whittaker Chambers testifies before the House Un-American Activities Committee, 1948. Photo: Bettman

In Horowitz’s case, this had ghastly consequences. His search for an agent of radical change in America had brought him into what Oppenheimer describes as “a close but fraught partnership” with the Black Panther leader Huey Newton. By the mid-1970s the Panthers had passed their peak. Many of their leaders were dead or in jail; Newton had fled to Cuba after being accused of killing a teenage prostitute. In November 1974, a young Panther was shot dead at a Panther-run school, meant to be a sanctuary for troubled children, which Horowitz had raised money to build. When he attended the memorial service he found Panther guards standing around the open casket, black berets aslant and shotguns held high. Faced with the jarring conjunction of children and guns, he began to realise that he was done with the revolutionary left. There was worse to come. Two months later, the body of Betty Van Patter, a Bay Area leftist he had recommended to the Panthers as a bookkeeper, was found washed up on the Bay shore. A blow to the head suggested she had died violently. The case was never solved, but Horowitz soon came to believe that the Panthers were responsible for her death – a view that later evidence supported. If this was so, he could not help thinking, then he, too, was to blame.

Re-evaluating his years with the Panthers, he accused himself of blanking out proof that casual violence had always been integral to the movement. Oppenheimer writes of Horowitz’s anger at himself and others, but surely there was much more to the change of heart. He came to see himself as having been possessed. “It was a terrible vision I had,” he wrote in his 1997 memoir, Radical Son. “I was basically repeating the mistakes of the previous generation of the left. I had sworn that I would never do that . . . I had schooled myself in Hegel and Marx, and where had they led me? I had worshipped the gods of reason, and they had delivered me into the company of killers.”

It’s an admission that is admirable for its unflinching honesty. The consequences for Horowitz included years of depression and heavy drinking, a failed marriage and the loss of many close friends. In May 1980 he was shaken by another funeral after Fay Stender, a left-wing criminal lawyer who had represented the Panthers, committed suicide. A year earlier, Stender had been shot and left partly paralysed by a member of the Black Guerrilla Family, a prison gang founded by a former client of hers. None of this was mentioned at her funeral service. Shocked and infuriated, Horowitz resolved to break the silence. In March 1981, his essay “Requiem for a Radical”, co-written with Peter Collier, was published in New West magazine. The piece was a ruthless dissection of the background and circumstances of Stender’s death, but the radical in question was as much Horowitz as the woman he mourned. After further soul-searching and a period investigating the activities of the Weather Underground, a clandestine direct-action group that had been engaged in a bombing campaign against public buildings, he concluded that his time on the left was definitively over. In 1984, “without hesitation”, he voted for Ronald Reagan.

Horowitz stands out among Oppenheimer’s subjects in having experienced the violent realities that are inseparable from revolutionary movements, but in this regard he has something in common with Whittaker Chambers: both are properly tragic figures. When, in 1932, Chambers agreed to do underground work for the Communist Party, first as a courier and then managing an elite cell of government officials in Washington, he did not ask exactly who or what he was working for. In fact, it was Soviet military intelligence. For his controllers the arrangement was productive, bringing them information passed on by US government personnel working in sensitive positions, including Harry Dexter White, who served as the most senior American representative at the 1944 Bretton Woods Conference that founded the postwar economic order, and Alger Hiss, who went on to represent the US at the Yalta Conference, where the fate of eastern Europe was settled.

In crossing the line into espionage, Chambers had made a fateful choice. In the late 1930s he began to have grave doubts about his decision when he read reports of Stalin’s purges of military leaders and party workers. Friends of his who went to the Soviet Union came back horrified and frightened, or, in some cases, they disappeared and were never heard from again. One who had criticised Stalin was abducted and murdered in New York City, and when Trotsky was assassinated in Mexico Chambers correctly predicted that the Soviet defector Walter Krivitsky, who died in a Washington hotel room in what had been made to look like suicide, would be next to be killed.

From seeing it as a remedy for the ills of the modern world, Chambers came to view communism as one of the worst modern disorders and resolved to break with it. Knowing that severing his ties with Soviet intelligence would be very dangerous, he began to take precautions, strapping a knife to his chest and setting up a safe house for himself and his family. He survived defection but his life was blighted. When in 1948 he testified in the trial against Hiss, whom he knew through his Communist networks, Chambers became a hate figure for liberals. He spent the rest of his days in relative obscurity. It’s a pity Oppenheimer doesn’t discuss these later years, when he was often thoughtful and courageous, opposing McCarthyism and the cult of Ayn Rand, whose cod-philosophy was gaining a growing following among sections of the right.

The things at stake for Chambers and Horowitz were matters of life and death, not fame and reputation. No such claim can be made for Norman Podhoretz (whose most painful experience, it seems, was having been omitted from Jackie Kennedy’s guest list) or Hitchens, who dissipated his undoubted brilliance in sulphurous polemics and flamboyant position-taking. Hitchens, with the quickest and most fertile mind of Oppenheimer’s sextet, at the same time displays most clearly the intellectual rigidity of the convert, continuing to defend the Iraq War even after many of those who engineered that ruinous adventure had had second thoughts. As Oppenheimer puts it, in one of the many pithy and unsparing assessments that pepper this acerbic and entertaining book, Hitchens (“equal parts Falstaff, Tom Paine, Tom Jones, Trotsky and Oscar Wilde”) became “an anachronism, a living but visibly dwindling remnant of a historical tendency – neoconservatism, liberal hawk-ism, Hitchens-ism – that had seemed vindicated, and had then been discredited, with tragic rapidity”.

A similar judgement must be made of the militant new conservatism that Oppenheimer’s political pilgrims helped create. When they moved rightwards it was not to emulate the caution and foresight exhibited by Dwight Eisenhower, still less the later tough-minded realism of George Bush, Sr. Instead, they renewed their delusions in another form. A “global democratic revolution” that would project Western institutions into vastly different societies was as much a fantasy as world communism. George W Bush’s crazed pursuit of regime change and its continuation in some policies of the Obama administration, particularly when acting under the direction of Hillary Clinton, were the result.

Except for Chambers and Horowitz, Oppenheimer’s apostates learned very little from their journey across the political spectrum. Those who banged the drum for war were as ignorant of the countries whose governments they wanted to overthrow as they been had of the workers they had claimed to be fighting for in the past. In both cases they used people of whom they knew nothing to satisfy their own need for significance. Believing they had left behind the mistakes of the radical left, they helped create a new right that repeated the same follies. Along the way, an older and more civilised conservatism was consigned to the memory hole.