The historian Eric Hobsbawm, who was born in 1917, the year of the Russian Revolution, and died in 2012 at the age of 95, was widely regarded as an unrepentant Stalinist, a man who, unlike other Marxist historians such as EP Thompson and Christopher Hill, never resigned his membership of the Communist party, and never expressed any regret for his commitment to the communist cause.

In the later part of his long life he was most probably the world’s best-known historian, his books translated into more than 50 languages and selling millions of copies across the globe (about a million in Brazil alone, for example). Yet when the BBC invited him on to the radio programme Desert Island Discs in 1995, the presenter Sue Lawley addressed him distantly as “Professor Hobsbawm”, left his books more or less unmentioned, and focused her attention so unremittingly on his lifelong commitment to communism that the programme turned from the usual comfortable chat into a hostile interrogation.

Many of the reviews of his bestselling history of the 20th century, The Age of Extremes, a work translated into 30 languages, charged him with playing down the evils of Stalinism, and the influential anti-communist French historians Pierre Nora and François Furet were so successful in preventing its publication in France that it was eventually translated into French by an obscure publishing house based in Belgium.

Facebook Twitter Pinterest Hobsbawm’s problems with the BBC resurfaced after his broadcast in 1971 on why the US was losing the Vietnam war. Photograph: Bettmann Archive

The label “Stalinist” dogged Hobsbawm throughout his adult life and affected his career in numerous ways. Even before the cold war had properly started, it prevented him from getting a job with the BBC. In 1945, he applied for a full-time post making educational broadcasts to help servicemen adjust to civilian life after a long period in the forces. The BBC found him “a most suitable candidate”, but the appointment was swiftly vetoed by MI5. Hobsbawm was, it warned, “not likely to lose any opportunity he may get to disseminate propaganda and obtain recruits for the Communist party”.

By 1947 he had managed to secure employment as a lecturer in history at Birkbeck College, London, then something of a haven for leftwingers whose academic careers had got into difficulties because of their political views. Although he had produced some specialised academic articles, his other publication plans had been frustrated by the same kind of suspicion that had blocked his career in the BBC. In 1955 his book The Rise of the Wage-Worker was rejected on the recommendation of two anonymous academic reviewers who found it lacking in objectivity because it was Marxist. The book remains unpublished to this day.

Some of the prejudice against Hobsbawm was clearly based on a feeling that he was somehow not quite British

Despite his growing reputation as an economic historian, Hobsbawm was for a long time unable to obtain promotion at Birkbeck. His applications for academic positions in economic history at Oxford and Cambridge were turned down on political grounds. In 1972, his problems with the BBC resurfaced after his broadcast on why America was losing the Vietnam war, delivered as part of a series called A Personal View, ran into trouble because of its support for the Vietnamese cause. The Americans pressured the BBC into commissioning a rebuttal by a former British intelligence officer, who argued somewhat implausibly that America was not losing the war at all. Of course, hostility to communism was, as Hobsbawm himself pointed out, much milder in Britain than it was in, say, the US. Still, it had a clearly discernible effect on his career.

Was Hobsbawm really the dangerous communist, the Stalinist apologist, the unrepentantly hardline Marxist that so many have assumed him to be? A careful reading of his autobiography, Interesting Times, published in 2002, as well of his other published work, will do a lot to dispel this simplistic view. But it is in the vast mass of private papers, including diaries, letters and unpublished personal reminiscences that the real answers are to be found. They can be supplemented by other sources, including the many files MI5 kept on him for several decades. What is the story this material tells?

Some of the prejudice against Hobsbawm was clearly based on a feeling that he was somehow not quite British (in contrast to real enemies of the country such as the Cambridge spies, who were graduates of public schools and therefore above suspicion). Born in Alexandria, he had spent his childhood in Vienna. This aroused suspicion in establishment circles. He was also Jewish by origin, a further black mark against his reputation (one Special Branch report described his uncle Harry, with whom he lived for a time in his teens, as “a sneering, critical type of person, harsh of speech, half Jew in appearance, having a long nose, thinning hair and blue eyes”).

Facebook Twitter Pinterest A National Socialist demonstration in Berlin, 1931. Photograph: Imagno/Getty Images

It was, and still is, widely thought that Hobsbawm was a refugee who fled with his family from Germany to England in 1933 in order to escape Hitler. In fact his father was British, and so he was a British citizen by birth. His mother, a passionate anglophile and professional translator, insisted on English being spoken at their home in Vienna. He was known by his fellow pupils at school as “the English boy”. There is no doubt, however, about his early commitment to the communist cause. In 1931, when he was sent to live in Berlin with an uncle and aunt following the premature death of his parents (his father from a heart attack, his mother from tuberculosis), he encountered an overheated political atmosphere that presented the young with a stark choice between communism and nazism. As an English boy from a liberal Jewish family, Hobsbawm had no option but to commit himself to communism.

But there were other, more personal reasons for his choice, reasons that help explain why he never abandoned the communist ideals he acquired in Berlin. The genteel poverty in which he had grown up in Vienna, and the miserable financial position of his uncle in Berlin, who had lost his job in the Depression as a result of laws restricting the numbers of foreign employees in German businesses, contrasted sharply with the relative prosperity of his fellow pupils at his high school. He felt ashamed of his shabby appearance and the straitened circumstances in which he lived. “Only by turning this completely around,” he confessed to his diary, “and becoming proud of it did I conquer the shame.” The real attraction of the communists was that they turned poverty into a virtue.

In these, his teenage years, following the death of his parents, Hobsbawm was engaged in a desperate search for a sense of family and belonging that was only partially satisfied by living with his uncle and aunt. For a short while he found it in the unlikely form of the Scouts, but it was the communist movement that really met all these deep emotional needs. He read a few basic Marxist texts, involved himself in the activities of the Socialist School Students’ League, and took part in the last great Communist party demonstration in Berlin, on 25 January 1933. A few days later, Hitler was appointed chancellor. Life became increasingly dangerous for communists and Jews. But it was for economic rather than political reasons that Hobsbawm’s uncle Sidney decided to move the family to Britain, after the failure of yet another business venture, this time in Barcelona. So many of Hobsbawm’s relatives were failed businessmen that it is hardly surprising he saw little future in capitalism.

Hobsbawm began reading the major Marxist texts. 'Drown yourself in Leninism,' was his note to himself. 'Let it become your second nature'

The German Communist party had continued to grow even while, towards the end of 1932, the Nazis were starting to lose support. Here was a movement that had 100 representatives in the national legislature. When Hobsbawm encountered the Communist party of Great Britain, the contrast could not have been greater. With not a single MP in Westminster until 1935, and a membership that made it little more than a sect, it did not impress Hobsbawm in the least. What’s more, it was an aggressively working-class organisation that had no time for intellectuals. Writing his diaries at home, in German, every day, Hobsbawm concluded it was not for him. He had already decided quite self-consciously that, as he put it, “I am an intellectual through and through.” He was beginning to realise that he was unusually clever, but he was already haunted by the feeling that he was physically unattractive. His cousin Denis said to him brutally he was “ugly as sin, but you have a mind”. Hobsbawm began reading voraciously, week by week, all the major Marxist texts. “Drown yourself in Leninism,” was his note to himself. “Let it become your second nature.” After reading 12 pages of Lenin he noted in his diary: “Astonishing how that cheers me up and clears my mind. I was in a total good mood afterwards.” This is not the feeling that most people have after ploughing through works such as Materialism and Empirio-Criticism. As for Stalin, Hobsbawm barely mentioned him at all.

He was also a political realist. The only mass movement of the left in Britain in the 1930s was the Labour party, and so Hobsbawm shunned the Communists and volunteered to help Labour in the local elections of 1934 (as he did in the general election of 1945). And he was far from narrow in his interests, visiting the major London galleries and museums, and reading numerous works (in English, French and German) of fiction, poetry and drama, as well as developing an enthusiasm for jazz, at a time when it was anathematised by the official cultural policy of the Communist party.

Facebook Twitter Pinterest A tragic necessity? … The invasion of Hungary by the USSR in 1956 was backed by the British Communist party. Photograph: AP

This breadth of pursuits only increased after he became an undergraduate at King’s College, Cambridge, in 1936. It was at this point, finding a growing number of students committing themselves to communism as a result of the Labour party’s failure to support the republic in the Spanish civil war, that he finally joined the Communist party, in the form of the university’s Socialist Club. But he quickly became bored with the club’s political dogmatism. He found its activities and especially its regular Bulletin, which he was charged with editing, “barren”, so he abandoned it for the non-political student periodical the Granta, duly becoming its editor, too. Here he had the scope for writing about cinema, a particular passion, but also producing profiles of leading Cambridge characters and visiting politicians.

After the war, he continued as a member of the party and did some work in supporting sister parties in central Europe, at least until they began to fall victim to a ruthless process of Stalinisation at the end of the 1940s. But in truth, Hobsbawm never behaved as a Communist was supposed to. He was not an activist, he did not sell Communist party literature on the street corner and he wrote regularly for non-Communist (“bourgeois”) publications, earning the party’s disapproval. He confessed himself an “outsider in the movement”. His main focus was on the work of the Communist Party Historians’ Group (CPHG), a relatively short-lived organisation of the late 1940s and early 50s largely confined to “theoretical discussions”. MI5 operatives, monitoring bugged conversations at Communist party HQ in London, noted that despite its attempts to make him behave like a loyal member, one senior official complained that “he had just ceased to answer any letters!”

What Hobsbawm was committed to was an ideal of communism with a small “C”, an ideal he had imbibed as an adolescent more through reading the Marxist classics than through taking an active part in the real politics of the movement. He also remained convinced, as he had been in the 1930s, that Communists would have to cooperate with other leftist parties in the struggle for power: hence his enthusiasm for the French Popular Front, which established a socialist and liberal government in 1936 with the support of the Communist party.

In the 50s, however, the chances of collaborating were minimal. The British Communist party was obdurately Stalinist, and without mass support. As time went on, Hobsbawm’s disillusion with it steadily grew. How could he, for example, possibly support Stalin’s policies when these involved show trials of “cosmopolitans”, or in other words, Jewish members, in Czechoslovakia and other communist-dominated countries in eastern Europe? After all, he knew a good number of them, and was aware that they were innocent of the charges brought against them.

Not long after the death of Stalin in 1953 the international communist movement was plunged into a deep crisis. On 25 February 1956, at the 20th congress of the Communist party of the Soviet Union, Nikita Khrushchev denounced Stalin for the “cult of personality” that had grown up around him, and for the numberless murders and atrocities he had committed. As the contents of the speech found their way to the west, the British party leadership tried to ignore them. But in April 1956 the historians’ group, led by Hobsbawm, Thompson and Hill, issued a stinging rebuke to the party for its failure to express regret for its “past uncritical endorsement of all Soviet polities and views”. An impassioned debate broke out in the party periodical World News, with Hobsbawm in particular calling for an open confrontation with the party’s past, its errors and its lies. He demanded that it had to be open to democratic change from below; simply imposing a “party line” from above was unacceptable. He met with delaying tactics and obfuscation from the leadership.

The crisis deepened in October 1956 when a liberal communist government came to power in Hungary on the back of mass popular demonstrations after months of stonewalling by the Stalinist regime in Budapest. On 4 November, Moscow responded with a military invasion, suppressing the new regime and killing at least 2,500 Hungarians who had tried to resist. These events, Hobsbawm declared, shocked party intellectuals and “pierced the core of their faith and hope”. Attempting to avoid an open confrontation with the leadership in London, which had backed the invasion, he conceded that the invasion was “a tragic necessity” in view of the threat of a reactionary rightwing government taking over, but demanded that “the USSR should withdraw its troops as soon as this is possible”.

Facebook Twitter Pinterest Tony Blair, whom Hobsbawm called ‘Thatcher in trousers’. Photograph: Jane Bown/The Observer

A furious debate broke out within the party as the leadership refused to budge. “Hobsbawm,” a phone conversation monitored by MI5 recorded one member as saying, “wants to call for the overthrow of the leadership and a new policy”. His attitude towards the party leaders was described as “bellicose”. As leading historians such as Thompson resigned from the CPGB in despair, Hobsbawm demanded the right to form an inner-party opposition. One leading figure in the party called him “a dangerous character”. He and the other historians, said another, were “a clueless lot of scruffs, potentially quite dangerous”. The “freedoms” they demanded would lead to “party anarchy”. Hobsbawm responded by attacking the CPGB’s “monumental complacency”. The party refused to budge. He and the other historians were “backboneless and spineless intellectuals”. Hill and most of the others resigned, having achieved nothing.

Hobsbawm went with them in many respects, contributing to their periodical the New Reasoner and joining with them in the New Left Club, outside the party. George Matthews, editor of the party paper, the Daily Worker, declared that “it would, in his opinion, be a good thing” if they “provoked Hobsbawm to leave the party”. He was in any case “an outsider”. Hobsbawm was summoned to party headquarters and told that “they wanted him to remain in the party and not do things that might put him out of it. Hobsbawm,” reported the MI5 monitor, “had been frightfully upset, swearing that he never wanted to leave.”

The exchange was a revealing one. Once again, Hobsbawm’s deep emotional commitment to the ideals of communism, symbolised for him by his continued membership of the party, had come to the fore. While most intellectuals in the party had become communists as part of the struggle against fascism in the 1930s, and so, once the struggle had been won, did not find it difficult to leave, Hobsbawm’s commitment went much deeper. Yet the obdurate Stalinism of the British party now left him out in the cold.

Facebook Twitter Pinterest Antonio Gramsci, the Italian Marxist theoretician and politician. Photograph: Laski Diffusion/Getty Images

After the mid-50s, he gravitated towards the very different model being forged by the reform-oriented “Eurocommunist” parties of Spain and Italy. By the 80s, following the ideas of Antonio Gramsci, he came to believe that the British Labour party had to reach out for an alliance with elements in the middle classes, since the old working class on which it had for so long relied for its support was now in decline; otherwise democracy in Britain was doomed. Far from being a Stalinist, he had now become the prophet of New Labour. His ideas were taken up by Neil Kinnock when he became leader of the Labour party, and put into action by Tony Blair, though he later came to regret Blair’s failure to unravel the neoliberal policies implemented by the Conservatives in the 80s (“Thatcher in trousers” was his verdict on Blair).

How did all this affect his practice as a historian? Is there any connection between Hobsbawm’s communism and the global fame and success of his historical writings?

The first and perhaps the most important thing to note is that his historical work was never purely Marxist. Far from being a “central European intellectual”, as some have claimed, he was influenced above all by French intellectual ideas, particularly those of the group of historians associated with the periodical Annales. Hobsbawm’s mentor at Cambridge in the late 1930s and afterwards, the economic historian Mounia Postan, introduced him to the work of the Annales, inviting their leading figure Marc Bloch to Cambridge and sharing in many respects their belief in history as an all-encompassing discipline, dealing analytically not only with politics, economy and society but also with the arts and indeed all aspects of life in the past.

Hobsbawm deepened this acquaintance with the French historical school in the 1950s, when he spent long periods in Paris mixing with dissident leftwing intellectuals. His book, The Age of Revolution, published in 1962, clearly showed the influence of the Annales, as did its successors The Age of Capital and The Age of Empire. What gave his writing a particularly wide appeal was nevertheless its indebtedness to Marxist models of interpretation, deployed with clarity and power and illustrated with examples and evidence taken from an astonishingly wide range of sources in a variety of languages. Here, his deep reading of European literature, beginning in his teenage years, displayed its influence in a style that combined elegance and wit, engaging the reader in a way that no conventional Marxist exposition could ever manage to do.

Marxist ideas helped him to develop concepts that made sense of the inchoate material of history

At the same time, like the other English Marxist historians such as Thompson, Hobsbawm was intellectually liberated by his distancing from the British Communist party in 1956. From writing in the 1940s and early 1950s about the rise of the working class, he turned to studying marginal and deviant people in history, “primitive rebels”, millenarians, Luddites, bandits, seemingly irrational popular movements that in fact expressed a strong degree of rationality in their rebellion against the encroachments of capitalism on their way of life. Of course, he slotted them into a basically Marxist teleology (they were “primitive” rebels after all, unlike the supposedly sophisticated modern rebels of Marxist labour movements). Still, the sympathy with which he treated them was obvious to everyone who was able to read between the lines.

Marxist ideas gave his work a coherence and structure that merely empirical history was unable to achieve; they helped him to develop concepts that made sense of the inchoate material of history, and at the same time, because they were novel and controversial, provided topics for debates and discussions that are still going on among historians today – “the general crisis of the 17th century”, “the declining standard of living in the industrial revolution”, “social banditry”, “the invention of tradition”, “the long 19th century” and many more. At the same time, concepts and ideas never forced the basic evidence on to the sidelines. Where fact and interpretation clashed, Hobsbawm was almost always scrupulous enough to yield to fact, as, for example, in his abandonment of Marxist theories of imperialism in his book The Age of Empire. Neither as a communist intellectual nor as a practising historian was he ever a mere propagandist.

As for his confrontation with the communist past in the final two decades of his life, following the fall of the Berlin Wall, there is no sign that he concealed or passed over the many crimes and atrocities that had disfigured it. The shrill demands to repent and recant with which he was so frequently confronted deserve to be treated with contempt. Rather, what gives The Age of Extremes much of its fascination is the spectacle of a lifelong communist trying, often but not always successfully, to come to terms with the failure of the cause he had served for so long as an intellectual.

• Eric Hobsbawm: A Life in History by Richard J Evans is published by Little, Brown on 7 February (RRP £35). To order a copy for £30, go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £15, online orders only. Phone orders min. p&p of £1.99.