Tony Benn was one of the most mesmerising and divisive figures in the mainstream of postwar British politics. An establishment insider who became a rebellious leftwing outsider, a cabinet minister turned street protester and reviled prophet of capitalism's demise, he nonetheless managed in old age to become something of a national treasure. "It's because I'm harmless now," he would explain.

In the course of a 60-year career in public life which left a more lasting impact on the constitution than on the direction of governments or their policies – first as Anthony Wedgwood Benn, briefly as Viscount Stansgate, and from 1973 as plain Mr Benn – he was both loved and loathed in equal measure by countless voters who had never met him.

As such Benn stood in a long line of upper-class, nonconformist radicals with a moral crusader's unsettling zeal, as recognisably English as a character out of Anthony Trollope or even PG Wodehouse. His former Oxford tutor, later his Notting Hill neighbour and cabinet colleague, the cerebral Tony Crosland, was devoted to him even though he accused Benn – known to some intimates as "Jimmy" – of working so hard that "he creates endless crises". Crosland would say affectionately: "Nothing the matter with him except he's a bit cracked."

It was a lack of any direct dealings with their troublesome, often self-righteous colleague which characterised many of "Jimmy's" more ardent admirers in the opinion of detractors less forgiving than Crosland. To them he symbolised disastrous and rancorous splits in the 1970s and 80s, a decade of unrealistic self-indulgence made worse when fellow-leftwingers, Michael Foot and Neil Kinnock led the party. Between the left and Roy Jenkins's rightwing SDP split, it cost them power for 19 years.

Benn outside parliament in 1961. Photograph: Philip Jones Griffiths

Surviving Bennites and their leftwing allies in the unions and grassroots labour movement were quick to counter criticism with praise for his far-sighted warnings against globalisation or unaccountable corporate power and his resilient optimism, after Benn's death at 88 was announced on Friday. With the possible exceptions of Aneurin Bevan and Arthur Scargill on the left and Margaret Thatcher or Enoch Powell on the radical right, no mainstream postwar political figure aroused such partisan loyalty – or fear.

Benn diaries



Throughout his adult life Benn was also a prolific keeper of what became nightly diary notes, later tape recordings, the basis of eight very readable volumes of diaries, the last published in 2013 as A Blaze of Autumn Sunshine. They provided insights into both his happy family life – married for 50 years to Caroline, an American of similar outlook – and Benn's take on the politics of the day, both high and low, plus gossip. In old age, the diaries were augmented by live performance on stage and TV, where he was as much a hit in the Tory home counties as in Labour heartlands. Even his worst enemies did not deny he was an excellent mimic who could be very funny.

Tony Benn with his son Hilary in 2007. Photograph: Antonio Olmos

The final diary entry concludes on a characteristically upbeat note that (also characteristically) merged both streams of the narrative: "I'm now waiting for the great-grandchildren to come along, as I think every younger generation brings fresh ideas into the world." By this stage his son Hilary had achieved the rare dynastic feat of becoming a third generation Labour cabinet minister and one granddaughter, Emily Benn, had already been a Labour parliamentary candidate at 20.

Lasting impact



With hindsight Benn's most lasting impact on politics has been his leading role in giving party activists – even in a reluctant Conservative party – a role in choosing their leader and in making "jobs for life" MPs more accountable via routine reselection procedures. Potentially of more significance in the future is another of his campaigns, to introduce referendums, a device that had previously been despised in Britain as a tool of despots.

Not for the first or last time, the practical outcome of his initial success – the 1975 referendum on UK membership of the then European Economic Community (EEC) – proved less satisfactory to him. Despite his prediction of victory on the morning of polling day, the cross-party No campaign, in which Benn was one of seven cabinets ministers to oppose Harold Wilson's Yes campaign, lost by a margin of 2-1. Benn's switch from youthful pro-Europeanism to hostility remained unswerving until his death. For him, as for Tory Eurosceptics, the core issue was national sovereignty.

Tony Benn on an antiwar protest in 1998 against the bombing of Iraq. Photograph: Michael Stephens/PA

Among other successes he also brought about the right of hereditary peers such as himself to renounce their inheritance. Aware that he would be debarred from the Commons when his father died (Tony's admired older brother Michael had been killed in the war) he also sought a change in the law. A three-year battle from 1960-1963 saw an election court declare a Tory "win" in Bristol SE when the peerage rules had declared the seat vacant: as Labour's defiantly readopted candidate Benn had won more votes, but was debarred as Viscount Stansgate. The showdown ended in victory though – as often with Benn's campaigns – the law of unintended consequences kicked in. Within months of Benn's reform it had allowed the 13th Earl of Home to renounce his peerage too and succeed Harold Macmillan as Tory PM. He ran Labour's Wilson very close in the 1964 election.

The pattern would recur. In his wider political campaigns Benn was usually less successful, sometimes even counter-productive. British voters did not opt to reject nuclear weapons or US military suzerainty, to leave Europe or embrace the "siege economy" alternative to both postwar social democracy and Thatcherite free markets. It was a strategy which came to a head in the traumatic IMF loan crisis of 1976 when the cabinet's leftwing members opposed the Jim Callaghan/Denis Healey cuts but was itself divided. Years later the former chancellor Healey admitted he had been wrong: the IMF loan was not needed.

Ministerial career



Repeatedly defeated by events and the electorate over the big issues, Benn enjoyed lesser ministerial successes, including a key role as minister for technology (1966-70) with the merger of ailing car firms into British Leyland (1968). It was part of an interventionist strategy that saw a similar solution in ICL, the new national champion created from computer mergers. As postmaster general (1964-66) he oversaw the introduction of the Girobank and famously commissioned some stamp designs that did not include the Queen's head. She inspected them without complaint, but No 10 phoned with a veto as soon as Benn was back at his desk.

Benn at Glastonbury in 2008. Photograph: John Rahim/Rex

When energy secretary (1976-79) he obtained better terms from the new North Sea oil companies than his Tory predecessors. He had been switched from the industry brief (1974-76) after a series of clashes with colleagues and officials, not least over his support for workers co-ops as an alternative to closure. He later claimed that during the 1974 election Whitehall had prepared three files for their incoming minister: one Tory, one Labour, the third marked "Labour, if not Mr Benn."

Concorde man



Always a keen gadget man, as a precaution against hostile editing, Benn taped all media interviews from mid-career and once claimed to have overheard a TV studio technician say "Remind me, are we lighting for or against tonight?" The story illustrated both his cultivated sense of paranoia and his humour. But even his greatest gadget, the Anglo-French supersonic Concorde which Benn enthusiastically embraced as minister for technology in the late 60s, eventually crashed and burned like so much else. Cynics linked his enthusiasm for the project to the constituency jobs in Bristol which went with it.

Benn canvassing in Bristol in 1950. Photograph: PA

Yet the greatest paradox of his career was that his early reputation as a wholesome, media-savvy Labour moderate-cum-moderniser (he called himself "the Peter Mandelson of the 1959 election") could have put him on an irresistible course to become party leader had he not swung so sharply to the left in the 1970s. Civil servants, bankers and industrialists backed by the mainstream orthodox media ("like the power of the medieval church") all conspired to thwart an elected reformist government, he concluded. So did the top-down nature of party leadership. "Industrial democracy" via workers control – an old syndicalist impulse – would be the answer.

Catalyst for change



Friends said the workers occupation of the Upper Clyde Shipbuilders (UCS) yards in Glasgow in 1971 was the catalyst which convinced Benn that under-performing British industry needed more radical remedies than postwar Croslandite social democracy offered. A decade later newly elected François Mitterrand briefly flirted with a similar strategy in France. By that time the industrial militancy and social disorder of the 1978-79 "winter of discontent" had already delivered to Britain the Thatcherite option.

Tony Benn. Photograph: Murdo Macleod

Enemies such as the chief whip, Michael Cocks, viewed Benn as a cabinet member disloyally destablising his own minority government with barely-coded speeches and briefings. Cocks would later write that Benn threw his lot in with the hard left – including the highly sectarian Trotskyite Militant Tendency, "entryists" who had infiltrated Labour's ranks – only because he realised he could never win the leadership without wresting the election process away from Labour MPs alone. One result was the 1981 electoral college which Ed Miliband currently seeks to reform. Another was the Liberal-SDP alliance, now the Liberal Democrats and now in coalition with David Cameron. Activist participation in policy-making, another Bennite demand much changed in the Blair era, remains to this day.

Personal frustration



Whatever the mixture of motives the results were personally frustrating. After the disappointments of the Kinnock years, for which Benn took much blame, produced a pragmatic reaction which led to three Labour election wins under Tony Blair and disappointments of a different kind. By widespread consent Benn's only rival as a communicator with Labour voters has been Blair, whom Benn ended up having to denounce impotently from the sidelines.

Benn and Denis Healey at party conference in 1981. Photograph: Don McPhee/Guardian

The fateful turn came in 1981 when he insisted on challenging Denis Healey for deputy leader a year after Foot, the "unity candidate" narrowly beat Healey to succeed Callaghan. Foot himself and young leftwing allies such as Robin Cook begged him not to further divide an exhausted party, but Benn announced his campaign in the dead of night and, from his power base on the National Executive Committee (NEC), lost by a whisker only because Tribune Group MPs such as Kinnock, Jeff Rooker and Joan Lester abstained rather than support him. Tribune, which Benn had only lately joined, duly split to create the Campaign Group.

Cocks earned his revenge in 1983 when he saw off Benn in a redrawn boundaries fight over their seats in Bristol. Defeated in marginal Bristol East, the activists hero – usually topping polls of the period – was therefore unable to fight the vacant leadership contest. In the wake of Labour's traumatic 143-seat defeat – in which it only narrowly saw off the SDP threat – the leadership was won by Foot's protegé, Kinnock. Benn got back via a byelection in Chesterfield in 1984, but by then Kinnock was starting to turn the tide against the left's "impossiblism". One prime example had seen Benn make a speech to the 1980 conference in which he promised that a future Labour government would nationalise key industries, control capital, abolish the Lords and repatriate all powers from Brussels "within weeks" of taking office.

Miners' strike



By now Benn was supporting Sinn Féin and a united Ireland, seen by many as equivocal over intimidatory tactics within Labour ranks and elsewhere. When another NEC leftwinger, Michael Meacher, delivered the swing vote against supporting Scargill's strategy for the miners strike of 1984-5, he said "May God forgive you." Scargill duly lost. In 1988, Benn stood against Kinnock for party leader against strong advice ("even from my wife," he told friends) and won 11.4% of his beloved electoral college. It was slightly less than the 11.8% he had got from MPs in the 1976 contest against the most talented field ever - Callaghan, Foot, Healey, Jenkins and Crosland.

Benn discussing politics with Ralph Miliband in Chesterfield in 1988. Photograph: John Harris/reportdigital.co.uk

Ahead of the 1992 election in which Kinnock lost against all expectations, Benn, still a Tory bogeyman, campaigned to replace the monarchy with a written constitution for a "democratic, federal and secular commonwealth". It was music to the ears of republicans, but there were not enough of them. In 2001 he stood down in Chesterfield (which turned to the Lib Dems), wittily announcing he was leaving parliament "to spend more time on politics".

Though never a pacifist, Benn opposed the Falklands war (and was lectured by Margaret Thatcher, an exact contemporary who had ducked war service, as he had not), visited Saddam Hussein during the 1991 Gulf war to help release hostages, and opposed Nato's military intervention in Kosovo (1998-99). It was inevitable that he accepted the presidency of the leftwing Stop the War Coalition over Iraq in 2003 and again visited Baghdad. He took part in the million-strong London demo, as he continued to do into the Miliband era, sometimes carrying a deckchair into Hyde Park from his home nearby. No longer the focus he could be guaranteed affectionate attention. "He was so good at it he completely fooled people," frustrated old rivals, much less loved, would explain.

Improbable journey



At almost every turn it had been an improbable journey. Both Benn's grandfathers were Liberal MPs, as was his father, William Wedgwood Benn, though he defected to the rising Labour party and briefly served in the cabinets of both Ramsay MacDonald and – as Lord Stansgate – of Clem Attlee. A privileged child of the progressive upper-middle class, one influenced more by his mother's religious moral compass than by leftwing economics, he met VIPs (including Gandhi ), attended Westminster school and read PPE at Oxford before joining the BBC via wartime service as an RAF pilot officer in Africa.

Michael Foot and Benn in 1980. Photograph: Don McPhee

It was Tony Crosland who helped "Jimmy" to the 1950 byelection seat at Bristol SE. Initially he was a centre-right moderate in the 50s battles between the Bevanite left and Hugh Gaitskell – for whom Benn voted as leader in 1955. Even when both were ministers Benn the good mimic would ring him up pretending to be a post office engineer or voter. Later he would be accused of naively idealising the working class.

Yet clues to his future radicalism has been there from the start. Benn was fiercely anti-colonial, joined anti-nuclear CND early and abandoned Gaitskell during the leader's doomed attempt to abandon Clause IV, Labour's commitment to nationalisation. It was the exact battleground on which Blair fought, and won, 40 years later.

It was a favourite Benn maxim that "issues not personalities" matter in politics, though year after year his own vivid personality – complete with trademark pipe and mug of tea – undermined the assertion. In a managerial era where the ideological battles embodied by Thatcherism versus Bennery have lost potency, he was almost the last of a disappearing species.