Had Eric Hobsbawm died 25 years ago, the obituaries would have described him as Britain's most distinguished Marxist historian and would have left it more or less there. Yet by the time of his death at the age of 95, he had achieved a unique position in the country's intellectual life. In his later years he became arguably Britain's most respected historian of any kind, recognised if not endorsed on the right as well as the left, and one of a tiny handful of historians of any era to enjoy genuine national and world renown.

Unlike some others, Hobsbawm achieved this wider recognition without in any major way revolting against either Marxism or Marx. In his 94th year he published How to Change the World, a vigorous defence of Marx's continuing relevance in the aftermath of the banking collapse of 2008-10. What is more, he achieved his culminating reputation at a time when the socialist ideas and projects that animated so much of his writing for well over half a century were in historic disarray, and worse – as he himself was always unflinchingly aware.

In a profession notorious for microscopic preoccupations, few historians have ever commanded such a wide field in such detail or with such authority. To the last, Hobsbawm considered himself to be essentially a 19th-century historian, but his sense of that and other centuries was both unprecedentedly broad and unusually cosmopolitan.

The sheer scope of his interest in the past, and his exceptional command of what he knew, continued to humble many, most of all in the four-volume Age of... series, in which he distilled the history of the capitalist world from 1789 to 1991. "Hobsbawm's capacity to store and retrieve detail has now reached a scale normally approached only by large archives with big staffs," wrote Neal Ascherson. Both in his knowledge of historic detail and in his extraordinary powers of synthesis, so well displayed in that four-volume project, he was unrivalled.

Hobsbawm was born in Alexandria, a good place for a historian of empire, in 1917, a good year for a communist. He was second-generation British, the grandson of a Polish Jew and cabinet-maker who came to London in the 1870s. Eight children, who included Leopold, Eric's father, were born in England and all took British citizenship at birth (Hobsbawm's Uncle Harry in due course became the first Labour mayor of Paddington).

But Eric was British of no ordinary background. Another uncle, Sidney, went to Egypt before the first world war and found a job there in a shipping office for Leopold. There, in 1914, Leopold Hobsbawm met Nelly Gruen, a young Viennese from a middle-class family who had been given a trip to Egypt as a prize for completing her school studies. The two got engaged, but the first world war broke out and they were separated. The couple eventually married in Switzerland in 1916, returning to Egypt for the birth of Eric, their first child.

"Every historian has his or her lifetime, a private perch from which to survey the world," he said in his 1993 Creighton lecture, one of several occasions in his later years when he attempted to relate his own lifetime to his own writing. "My own perch is constructed, among other materials, of a childhood in the Vienna of the 1920s, the years of Hitler's rise in Berlin, which determined my politics and my interest in history, and the England, and especially the Cambridge, of the 1930s, which confirmed both."

In 1919, the young family settled in Vienna, where Eric went to elementary school, a period he later recalled in a 1995 television documentary which featured pictures of a recognisably skinny young Viennese Hobsbawm in shorts and knee socks. Politics made their impact around this time. Eric's first political memory was in Vienna in 1927, when workers burned down the Palace of Justice. The first political conversation that he could recall took place in an Alpine sanatorium in these years, too. Two motherly Jewish women were discussing Leon Trotsky. "Say what you like," said one to the other, "but he's a Jewish boy called Bronstein."

In 1929 his father died suddenly of a heart attack. Two years later his mother died of TB. Eric was 14, and his Uncle Sidney took charge once more, taking Eric and his sister Nancy to live in Berlin. As a teenager in Weimar Republic Berlin, Eric inescapably became politicised. He read Marx for the first time, and became a communist.

He could always remember the day in January 1933 when, emerging from the Halensee S-Bahn station on his way home from his school, the celebrated Prinz Heinrich Gymnasium, he saw a newspaper headline announcing Hitler's election as chancellor. Around this time he joined the Socialist Schoolboys, which he described as "de facto part of the communist movement" and sold its publication, Schulkampf (School Struggle). He kept the organisation's duplicator under his bed and, if his later facility for writing was any guide, probably wrote most of the articles too. The family remained in Berlin until 1933, when Sidney Hobsbawm was posted by his employers to England.

The gangly teenage boy who settled with his sister in Edgware in 1934 described himself later as "completely continental and German speaking". School, though, was "not a problem" because the English education system was "way behind" the German. A cousin in Balham introduced him to jazz for the first time – the "unanswerable sound", he called it. The moment of conversion, he wrote some 60 years later, was when he first heard the Duke Ellington band "at its most imperial". He spent a period in the 1950s as jazz critic of the New Statesman, and published a Penguin Special, The Jazz Scene, on the subject in 1959 under the pen-name Francis Newton (many years later it was reissued with Hobsbawm identified as the author).

Learning to speak English properly, Eric became a pupil at Marylebone grammar school and in 1936 he won a scholarship to King's College, Cambridge. It was at this time that a saying became common among his Cambridge communist friends: "Is there anything that Hobsbawm doesn't know?" He became a member of the legendary Cambridge Apostles. "All of us thought that the crisis of the 1930s was the final crisis of capitalism," he wrote 40 years later. But, he added, "it was not."

When the second world war broke out, Hobsbawm volunteered, as many communists did, for intelligence work. But his politics, which were never a secret, led to rejection. Instead he became an improbable sapper in 560 Field Company, which he later described as "a very working-class unit trying to build some patently inadequate defences against invasion on the coasts of East Anglia". This, too, was a formative experience for the often aloof young intellectual prodigy. "There was something sublime about them and about Britain at that time," he wrote. "That wartime experience converted me to the British working class. They were not very clever, except for the Scots and Welsh, but they were very, very good people."

Hobsbawm married his first wife, Muriel Seaman, in 1943. After the war, returning to Cambridge, he made another choice, abandoning a planned doctorate on north African agrarian reform in favour of research on the Fabians. It was a move that opened the door to both a lifetime of study of the 19th century and an equally long-lasting preoccupation with the problems of the left. In 1947 he got his first tenured job, as a history lecturer at Birkbeck College, London, where he was to remain for much of his teaching life.

With the onset of the cold war, a very British academic McCarthyism meant that the Cambridge lectureship which Hobsbawm always coveted never materialised. He shuttled between Cambridge and London, one of the principal organisers and driving forces of the Communist Party Historians Group, a glittering radical academy which brought together some of the most prominent historians of the postwar era. Its members also included Christopher Hill, Rodney Hilton, AL Morton, EP Thompson, John Saville and, later, Raphael Samuel. Whatever else it achieved, the CP Historians Group, about which Hobsbawm wrote an authoritative essay in 1978, certainly provided a nucleus for many of his first steps as a major historical writer.

Hobsbawm's first book, Labour's Turning Point (1948), an edited collection of documents from the Fabian era, belongs firmly to this CP-dominated era, as does his engagement in the once celebrated "standard of living" debate about the economic consequences of the early industrial revolution, in which he and RM Hartwell traded arguments in successive numbers of the Economic History Review. The foundation of the Past and Present journal – now the most lasting, if fully independent, legacy of the Historians Group – also belongs to this period.

Hobsbawm was never to leave the Communist party and always thought of himself as part of an international communist movement. For many, this remained the insuperable obstacle to an embrace of his writing. Yet he always remained very much a licensed free-thinker within the party's ranks. Over Hungary in 1956, an event which split the CP and drove many intellectuals out of the party, he was a voice of protest who nevertheless remained.

Yet, as with his contemporary, Christopher Hill, who left the CP at this time, the political trauma of 1956 and the start of a lastingly happy second marriage combined in some way to trigger a sustained and fruitful period of historical writing that was to establish fame and reputation. In 1959 he published his first major work, Primitive Rebels, a strikingly original account, particularly for those times, of southern European rural secret societies and millenarian cultures (he was still writing about the subject as recently as 2011). He returned to these themes again a decade later in Captain Swing, a detailed study of rural protest in early 19th-century England co-authored with George Rudé, and Bandits, a more wide-ranging attempt at synthesis. These works are reminders that Hobsbawm was both a bridge between European and British historiography and a forerunner of the notable rise of the study of social history in post-1968 Britain.

By this time, though, Hobsbawm had already published the first of the works on which both his popular and academic reputations still rest. A collection of some of his most important essays, Labouring Men, appeared in 1964 (a second collection, Worlds of Labour, was to follow 20 years later). But it was Industry and Empire (1968), a compelling summation of much of his work on Britain and the industrial revolution, that achieved the highest esteem. It has rarely been out of print.

Even more influential in the long term was the Age of… series, which he began in 1962 with The Age of Revolution: 1789-1848. This was followed in 1975 by The Age of Capital: 1848-1875 and in 1987 by The Age of Empire: 1875-1914. A fourth volume, The Age of Extremes: 1914-91, more quirky and speculative but in some respects the most remarkable and admirable of all, extended the sequence in 1994.

The four volumes embodied all of Hobsbawm's best qualities – the sweep combined with the telling anecdote and statistical grasp, the attention to the nuance and significance of events and words, and above all, perhaps, the unrivalled powers of synthesis (nowhere better displayed than in a classic summary of mid-19th century capitalism on the very first page of the second volume). The books were not conceived as a tetralogy, but as they appeared, they acquired individual and cumulative classic status. They were an example, Hobsbawm wrote, of "what the French call 'haute vulgarisation'" (he did not mean this self-deprecatingly), and they became, in the words of one reviewer, "part of the mental furniture of educated Englishmen".

Hobsbawm's first marriage had collapsed in 1951. During the 1950s, he had another relationship which resulted in the birth of his first son, Joss Bennathan, but the boy's mother did not want to marry. In 1962 he married again, this time to Marlene Schwarz, of Austrian descent. They moved to Hampstead and bought a small second home in Wales. They had two children, Andrew and Julia.

In the 1970s, Hobsbawm's widening fame as a historian was accompanied by a growing reputation as a writer about his own times. Though he had a historian's respect for the Communist party's centralist discipline, his intellectual eminence gave him an independence that won the respect of communism's toughest critics, such as Isaiah Berlin. It also ensured him the considerable accolade that not one of his books was ever published in the Soviet Union. Thus armed and protected, he ranged fearlessly across the condition of the left, mostly in the pages of the CP's monthly, Marxism Today, the increasingly heterodox publication of which he became the house deity.

His conversations with the Italian communist – and now state president – Giorgio Napolitano date from these years, and were published as The Italian Road to Socialism. But his most influential political work centred on his increasing certainty that the European labour movement had ceased to be capable of bearing the transformational role assigned to it by earlier Marxists. These uncompromisingly revisionist articles were collected under the general heading The Forward March of Labour Halted.

By 1983, when Neil Kinnock became the leader of the Labour party at the depth of its electoral fortunes, Hobsbawm's influence had begun to extend far beyond the CP and deep into Labour itself. Kinnock publicly acknowledged his debt to Hobsbawm and allowed himself to be interviewed by the man he described as as "my favourite Marxist". Though he strongly disapproved of much of what later took shape as "New Labour", which he saw, among other things, as historically cowardly, he was without question the single most influential intellectual forerunner of Labour's increasingly iconoclastic 1990s revisionism.

His status was underlined in 1998, when Tony Blair made him a Companion of Honour, a few months after Hobsbawm celebrated his 80th birthday. In its citation, Downing Street said Hobsbawm continued to publish works that "address problems in history and politics that have re-emerged to disturb the complacency of Europe".

In his later years, Hobsbawm enjoyed widespread reputation and respect. His 80th and 90th birthday celebrations were attended by a Who's Who of leftwing and liberal intellectual Britain. Throughout the late years, he continued to publish volumes of essays, including On History (1997) and Uncommon People (1998), works in which Dizzy Gillespie and Salvatore Giuliano sat naturally side by side in the index as testimony to the range of Hobsbawm's abiding curiosity. A highly successful autobiography, Interesting Times, followed in 2002, and Globalisation, Democracy and Terrorism in 2007.

More famous in his extreme old age than probably at any other period of his life, he broadcast regularly, lectured widely and was a regular performer at the Hay literary festival, of which he became president at the age of 93, following the death of Lord Bingham of Cornhill. A fall in late 2010 severely reduced his mobility, but his intellect and willpower remained unvanquished, as did his social and cultural life, thanks to Marlene's efforts, love – and cooking.

That his writings continued to command such audiences at a time when his politics were in some ways so eclipsed was the kind of disjunction which exasperated rightwingers, but it was a paradox on which the subtle judgment of this least complacent of intellects feasted. In his later years, he liked to quote EM Forster that he was "always standing at a slight angle to the universe". Whether the remark says more about Hobsbawm or about the universe was something that he enjoyed disputing, confident in the knowledge that it was in some senses a lesson for them both.

He is survived by Marlene and his three children, seven grandchildren and one great-grandchild.

• Eric John Ernest Hobsbawm, historian, born 9 June 1917; died 1 October 2012

• Dorothy Wedderburn died on 20 September 2012