When the Soviet ambassador Ivan Maisky reported to Beatrice Webb that Winston Churchill had told him, “Better communism than Nazism!” she was not surprised. Views of this sort were not untypical of the British ruling elite. But Churchill did not really belong in that class. “Churchill is not a true Englishman, you know,” the celebrated social scientist said. “He has negro blood. You can tell even from his appearance.” Webb went on to recount to Maisky “a long story” about Churchill’s mother coming from the American South and her sister looking just like a “negroid”.

Possibly aiming to change the subject, Maisky mentioned Henry Stanley, the explorer of Africa. At this point Webb became “agitated”, and began talking about Stanley’s marriage to a “beautiful young girl” who came “from a very good family”, while Stanley was a “real upstart, a coarse, uncouth fellow”. In support of this judgement Webb appealed to her husband, Sidney, “whose expression and gestures indicated full assent”. Maisky concludes his account of the conversation with the comment: “The crux of the matter is that Stanley was a true plebeian, and this matters.”

Maisky recorded the episode in the journals he kept of his 11 years as ambassador to Britain, edited by Gabriel Gorodetsky and published as The Maisky Diaries: Red Ambassador to the Court of St James ­(1932-1943). Maisky was familiar with the quirks of the progressive mind. A former Menshevik, he was never trusted by the Bolsheviks, or by Stalin. Born in 1884 and growing up in the Siberian city of Omsk, where his Polish-Jewish father worked as a medical officer, he was the target of anti-Semitic prejudice both in Russia and in Britain. At the Foreign Office he was described as “that little Tartar Jew”, and not long before Stalin’s death in 1953 he was charged with involvement in a “Zionist plot” and imprisoned. Released in 1955, he spent his remaining years in obscurity and died in 1975.

If Stalin had lived longer, Maisky might have joined the millions who perished in Soviet labour camps. Stalin’s regime had been eulogised by the Webbs in their book The Soviet Union: a New Civilisation?, published in 1935, after visits to the USSR from which they returned gushing with enthusiasm. In this they were not unusual. Throughout the 1930s – the most savage and bloodstained period in the history of the Soviet state, though Stalin’s methods were only those of Lenin applied on a larger scale – streams of Western fellow-travellers went to the Soviet Union and came back convinced that it embodied humankind’s best hopes for the future. By way of testimony to this ardent faith, the Webbs removed the question mark from later editions of the book.

The Webbs, who founded the New Statesman, had never been communists and had no great sympathy with Marxism. Sidney, a founding member of the Fabian Society, became a Labour MP and served as secretary of state for the colonies in Ramsay MacDonald’s government. Beatrice, the daughter of a wealthy businessman, inherited a substantial private income and thought of herself as belonging to a class that “habitually gave orders”. The largest intellectual influence on her was Herbert Spencer, the prophet of laissez-faire capitalism, who visited her family’s home often and for whom she worked for a time as an assistant. An apostle of the idea of social evolution, Spencer believed the world was progressing towards individualism and minimal government; in his final years, when this failed to occur, he fell into depression. Sidney and Beatrice were also believers in social evolution, but for them it had a different direction. They had no doubt that Soviet collectivism represented the next phase in history.

The gaggles of bien pensant writers and journalists, liberal teachers and academics, radical aristocrats and businessmen who flocked to the Soviet Union and later Mao’s China went to these countries convinced that their own societies were stuck in the past. They believed that only a thinking minority – themselves – could see the outlines of a better future. Plainly, it was these advanced minds that could direct the new society that was coming into being.

The Webbs took it for granted that when socialism came to Britain, they and people like them would still be giving the orders. Anthony Eden – whom Maisky reports as saying that socialism was inevitable – assumed the same. A belief that they would form part of the coming ruling class inspired the Cambridge spies. Describing his recruitment as an agent of the NKVD (later KGB), Kim Philby wrote that he did not hesitate when offered “enrolment in an elite force”. Throughout his long exile in the Soviet Union, Guy Burgess wore an Old Etonian tie along with the Order of the Red Banner that he had received for his services to the Soviet state. For these and others in the upper reaches of British life in the 1930s, it was clear that British imperial power was in decline. Identifying themselves with the Soviet cause was a way of securing their place in the new world order.

Eugene Lyons, a left-wing American journalist who spent the early 1930s working as a Western correspondent in Moscow, wrote in his forgotten autobiographical masterpiece Assignment in Utopia (1937) that the Americans who visited the Soviet Union during those years fell into distinct groups. Some had “a professional interest” in being on amicable terms with agents of the state. Walter Duranty, the British-born Moscow correspondent of the New York Times, used his columns to ridicule the notion that there was famine anywhere in the Soviet Union, while confiding to officials at the British embassy that about ten million may have died of starvation. Described by Lyons as being “curiously contemptuous” of ordinary Russians, Duranty was provided by the Soviet state with a spacious apartment, a large car, sumptuous meals and the services of attractive women, “former persons” from the old regime who had been coerced into working for the secret police. For Duranty, fellow-travelling was a ticket to a kind of life he could not have enjoyed in a Western country.

Others used their visits to showcase their intellect and wit. Lyons reports how in 1931 George Bernard Shaw celebrated his 75th birthday at a banquet in Moscow, which he visited that summer with Nancy, Lady Astor. Giving a speech, Shaw recalled that when his friends heard he was going to the Soviet Union they loaded him with tinned food in the absurd belief that Russia was starving: “But I threw all their food out of the window in Poland before I reached the Soviet frontier.” He had been overstuffed ever since he reached Moscow, he joshed. Shaw’s listeners gasped, then managed a little forced laughter. Even the members of the Soviet elite who had been assembled to listen to the old prankster knew how scarce food had become. “At this gathering,” Lyons writes, “Shaw achieved the apex of cynicism. In any other man it might have been ignorance or stupidity; in Shaw it was a cold and calculating taunting of the audience.”

And yet the largest numbers of fellow-travellers coming to the Soviet Union were far from being cynics. Most belonged to the category Lenin is reputed to have called “useful idiots”: “Professors or liberal clergymen, they were deeply disturbed by the shattered economic and social orthodoxies in which they were raised; if they lost their compensating faith in Russia life would ­become too bleak to endure.” The Great Depression had destroyed their belief in the ideas that governed their own societies, and they desperately needed a surrogate creed. When they travelled to the Soviet Union, it was a pilgrimage of faith.

The great majority of these pilgrims closed their eyes to signs of brutal repression which – despite being cosseted guests of the Soviet state – they must have glimpsed at some point during their stay. Describing the fate of the peasants, Lyons writes:

A population as large as all of Switzerland’s or Denmark’s was stripped clean of all their belongings – not alone their land and homes and cattle and tools, but often their last clothes and food and household utensils – and driven out of their villages. They were herded with bayonets at railroad stations, packed indiscriminately into cattle cars and freight cars, and dumped weeks later in the lumber regions of the frozen North, the deserts of Central Asia, wherever labour was needed, there to live or die . . . The spectacle of peasants being led by soldiers with drawn revolvers through the streets even of Moscow was too commonplace to win more than a casual glance from the crowds on the sidewalks.

Somehow, most Western visitors seem not to have witnessed these everyday scenes – or if they did they soon managed to forget them. Impelled by a sense of honesty or shame, some later reported what they had seen. Arthur Koestler, who had turned to the Soviet Union as embodying a semi-mystical vision of order and a bulwark against fascism, recounted in The ­Invisible Writing (1954) what he had witnessed when travelling in Ukraine, where he lived for some months in 1932 and 1933 and where he saw the effects of the famine Stalin had engineered in that country:

The train puffed slowly across the Ukrainian steppe . . . At every station there was a crowd of peasants in rags, offering ikons and linen in exchange against a load of bread. The women were lifting up their infants to the compartment windows – infants pitiful and terrifying with limbs like sticks, puffed bellies, big cadaverous heads lolling on thin necks. I had arrived, unsuspecting, at the peak of the famine of 1932-33 which had depopulated entire districts and claimed several million victims.

Very few fellow-travellers recorded such scenes as Koestler did in his superb memoir. An “inner censor”, he believed, stood in the way. After all, what were these miserable peasants anyway? It was irrational sentimentality to waste one’s tears on these ignorant, superstitious and obsolete specimens when a new humanity was being fashioned. ­Koestler soon recognised that this was a fantasy and he left the British Communist Party in 1938. Other fellow-travellers persisted in their quest, some moving on to other countries to salvage their illusions.

More than any particular shift in allegiance, fellow-travelling is a frame of mind – an urgent need to believe that a new and far better kind of society is emerging in some other land. Not all progressive thinkers have succumbed to this weakness. When Bertrand Russell visited Russia to meet Lenin, he quickly perceived that the regime the Soviet leader was building was intrinsically despotic. Writing on his return in The Theory and Practice of Bolshevism (1920), a remarkably prescient book, Russell identified the chief source of Soviet repression as being not the exigencies of the civil war nor the continuing influence of tsarist authoritarianism, but Bolshevism itself: “A great part of the despotism that characterises the Bolsheviks belongs to the essence of their social philosophy, and would have to be reproduced, even if in a milder form, wherever that philosophy became dominant.” He concluded: “Bolshevism as a social phenomenon is to be reckoned as a religion, not an ordinary political movement.”

These observations did not make Russell popular with his fellow progressives, by whom he was mistrusted for many years. He was pointing to a fact that most progressives refuse to confront to this day: that the regime Lenin founded engaged in systematic repression from the beginning.

***

The Declaration of the Rights of the Toiling and Exploited Peoples, promulgated in January 1918 as one of the founding documents of the Soviet state, created the category of “former persons” – class enemies who were to be deprived of civil rights, including entitlement to food rations. These retrograde human types and anyone economically dependent on them – in other words, their families – had to be excluded from the new society that was under construction. The summary executions of landlords, priests, sex workers and other obsolete groups that followed served the same imperative. When Lenin ordered the execution of refractory peasants in his “Hanging Order” of August 1918, instructing that they be killed “in full view of the people”, so that everyone “for hundreds of miles around will see, tremble, know and scream out”, he had no doubt that this was a necessary condition of human progress.

So was crushing the large-scale peasant rebellion in the Tambov region several hundred miles south-west of Moscow in 1920-21. Peasant forces in the area had fought against the Whites during the civil war. This did not stop the Red Army destroying whole villages, using poison gas to flush out villagers from the forests where they had fled and herding women, children and old people who survived the attacks into camps, where many of them died. Accounts of these events were available in Russian émigré publications and occasionally surfaced in the Western press, but they failed to dent the image of the Bolshevik regime. If Western progressives were aware of such atrocities, they explained them as showing the new regime’s need to defend itself against the forces of reaction.

The ease with which fellow-travellers pass over the casualties of the regimes with which they identify is one of their defining traits. The most inveterate among them do not limit their enthusiasm to any single state. They are attracted by any large political experiment that seems to prefigure a new order of things. Not long after accompanying Shaw on his frolics in the Soviet Union, Nancy Astor emerged as a moving force behind the set of pro-Nazi appeasers that frequently gathered at her stately home in Cliveden, Buckinghamshire.

On the European continent, ­intellectuals who had been communist sympathisers found reasons in favour of the new European order that they saw emerging in the late 1930s. One of the precursors of post-structuralism, the French literary critic Georges Bataille, who some years earlier had been a member of a dissident communist group, delivered a lecture in 1939 in Paris entitled “Hitler and the Teutonic Order”. There is nothing to suggest Bataille was being ironic.

Fellow-travellers may sometimes look like opportunists, but it is opportunism of a particular kind – the belief that the regime with which they identify is being propelled by irresistible historical forces. Yet history mocks all such certainties. Those who embraced the Soviet cause in the 1930s did so in the conviction that capitalism was doomed, along with nationalism and religion. In the introduction to his 1968 account of his years as a Soviet agent, My Silent War, Kim Philby wrote complacently: “As I look over Moscow from my study window, I can see the solid foundations of the future I glimpsed at Cambridge.” Philby died in May 1988. Just a few years later, the solid foundations he admired from his study window had collapsed. Blending authentically Bolshevik methods of government-by-fear with crony capitalism, ethnic nationalism and resurgent religious Orthodoxy, the regime that emerged from the ruins would have been inconceivable to him.

Fellow-travelling did not end with the fall of communism. Many who had believed in the Soviet Union latched on to other states they supposed were in the vanguard of history. For some former Trotskyites, the United States became the emancipatory power they believed the USSR could once have been. According to the late Christopher Hitchens, the invasion of Iraq was only the first in a succession of revolutionary wars of liberation. Possessed by the fantasy that another new world was in the making, former leftist visionaries such as Hitchens became the Bush administration’s useful idiots.

As some of us warned in this paper before the invasion began, toppling Saddam Hussein would lead to the break-up of the state of Iraq and an explosive increase in support for radical Islamist forces. But Iraq was attractive to some who had been on the far left as a laboratory for a political experiment of a kind that had failed elsewhere. US-style democracy would be installed, not only in the Middle East, but in countries throughout the world. Since then, the United States has undergone an unexpected mutation. Today the standard-bearers of a new world order are chafing under the whimsical rule of a would-be despot. The US system of government is struggling to survive on its home ground.

***

Given the powerful psychological needs it serves, fellow-travelling is unlikely ever to disappear completely. True, there are not many candidates for a new civilisation at present. For some, “Europe” – the ethereal project, not the diverse and interesting continent – has replaced visions of socialism; but European institutions are in disarray, possibly disintegrating and at best stagnant and immobile. Some enlightened circles appear attracted by Norwegian petro-egalitarianism; but Norway is small, dull and lacking in the violence that fellow-travellers seem to crave. It may be no accident that many of them lost interest in Russia when the Soviet state became less repressive after Stalin’s death. Some transferred their allegiance to Mao’s China, where the Great Leap Forward and the Cultural Revolution answered to a need the Soviet Union no longer satisfied. When the frenzy that fuelled these upheavals passed, many lost interest in China as well.

Russia remains perennially fascinating. But those on the left who defend Vladimir Putin do so more through hatred of their own societies than any virtues they discern in the regime over which he presides, while liberals who rage against him seem more indignant about his crimes than they were about any perpetrated by his communist predecessors. Both groups seem to feel a certain nostalgia for the former Soviet Union,

which may have killed far more people but did so for, as they see it, progressive reasons.

A wistful yearning for revolutionary violence may be one reason for the rise of a later generation of fellow-travellers who identify with Islamism. Apart from the hereditary dictatorship that governs North Korea and a failed state in Venezuela, there are no revolutionary regimes left in the world. Islamist movements fill this gap by combining hatred of the West with Leninist methods of remodelling society by force – a mix that some on the left evidently find appealing. But the glory days of fellow-travelling are gone. One benefit of globalisation is that nobody can imagine any longer that a utopian society is coming into being in some remote corner of the planet.

No doubt there will be some who regret the passing of the large political hopes that fellow-travelling once expressed. But this loss afflicts the privileged groups from which fellow-travellers are commonly drawn more than the shadowy and mute figures for whom they claim to speak. The populations of the countries that fellow-travellers visited were as alien and unknown to them as the “negroids” whose ancestral traces the Webbs and others detected in Winston Churchill.

The principal goal of fellow-travellers has always been to sustain their own sense of having a special place in history. The millions whose lives were destroyed in the course of grandiose political experiments that led nowhere served a cause of which they were unaware – keeping up the spirits of the Western thinking classes.

John Gray’s books include “The Soul of the Marionette: a Short Inquiry into Human Freedom” (Penguin)