It was in my journalistic infancy that I first interviewed Tony Benn. He was highly courteous. Endless mugs of tea were offered. Tobacco smoke belching from his pipe, he seemed quite avuncular. His conversation was never less than stimulating. This audience took place at a transitional stage of his career: he was still a fearful bogeyman to the right and mistrusted when not intensely hated by the many Labour colleagues who blamed him for splitting and nearly destroying the party; the transformation from "most dangerous man in Britain" to "national treasure" lay in the future, but the zenith of his power had passed. The Bennite ascendancy of the early 1980s, when he and his fellow travellers on the extreme left came close to seizing control of the commanding heights of Labour, was over. Neil Kinnock had embarked on the long march back to electability, which would remorselessly push him to the margins of his party.

Interviewing Mr Benn, even a diminishing Mr Benn, was nevertheless an intimidating experience for a young reporter, not least because when I set up my tape machine, he plonked down a recording device of his own. At the time, this was unusual behaviour by a politician, which some would put down to his well-developed, though not entirely unjustified, paranoia about the media. I came away with an interview I thought interesting – and he made no complaint about how I used it. That was a more fortunate experience than a colleague. An interview he conducted did not go to the liking of Mr Benn. So much not to his liking that he suddenly drew from a cupboard a magnetic device and waved it over the reporter's machine, erasing the tape.

I tell this story to illustrate a couple of things about him that were ahead of his time. These days, all politicians make recordings of an interview as a precaution against being misrepresented. In this respect, if not many others, they are all Bennites now. Another thing it demonstrates is his lifelong love of gadgets. The basement of his large house in Holland Park was packed with his vast archive, a museum of socialist memorabilia and a collection of machines of various vintages, including those on which he recorded his entertaining and revealing diaries. Late in life, he tried his hand at entrepreneurship by promoting an invention called the "seatcase", a piece of luggage for the elderly that doubled up as something you could sit on. I believe it never quite made it to The Dragons' Den.

Though he became the fiercest opponent of Labour "modernisation", wrongly predicting that it would never win any elections, he was in some ways the grand-daddy of them all. One of the first politicians to master television, by his own estimation Tony Benn was "the Peter Mandelson of the 1959 election". He was the frontman for the early Harold Wilson's attempt to move Labour on from class conflict by extolling "the white-hot heat of the technological revolution". As a middle-of-the-road minister in the 1960s, he rather excelled at self-promotion and what we would now call spin. As postmaster-general, he oversaw the opening of the Post Office Tower, then the tallest building in Britain and with a fancy, revolving restaurant offering panoramic views of the capital to those who could afford the menu. As minister of technology, he sponsored Concorde, a very fast plane for very rich people.

It was not until Labour's later period of power in the 1970s that he migrated leftwards. For some, this was a refreshing and rare example of a politician becoming more radical with experience of power. For others, it was an example, in the words of Wilson, of someone who "immatures with age".

This reinvention as a champion of the working class was accompanied by a mildly comic effort to airbrush away the elite elements of his biography. He deleted from Who's Who any reference to his private education at Westminster School. He made a formal announcement declaring that he no longer wished to be known as Anthony Wedgwood Benn, but as the more proletarian Tony. I have often thought that it was as much this as it was fear of his views that aroused such venom against him on the right. As they saw it, he was not only threatening revolution, the former Viscount Stansgate was a traitor to his class. The campaign to be allowed to renounce his hereditary peerage in order to remain an MP led to the Peerage Act of 1963. As was often the case with him, it had consequences that he had not intended. It opened the door for Alec Douglas-Home to succeed Harold Macmillan as Tory prime minister.

Another, more important constitutional innovation was to introduce the concept of the referendum into British politics with consequences that reverberate to this day. When the Labour party was riven with division over membership of what was then the Common Market, he was the first significant figure to push for a plebiscite as a means to resolve the split. Again, it did not have the outcome that he, who had become an opponent, had expected. The referendum affirmed British membership by a margin of two to one.

His third contribution to the way we do politics was to campaign for MPs to lose their exclusive franchise in party leadership elections. All the parties have since adopted versions of one member, one vote. Yet this, too, did not have the result that he anticipated. After his challenge to Denis Healey for the deputy leadership was beaten off by an eyebrow, his prospects of becoming leader receded to vanishing point.

In the honeyed flow of eulogising that has marked his death, Ed Miliband has described him as "an iconic figure" and David Cameron as "a magnificent campaigner". The hard truth that both were too polite to mention is that he was a failure as a practical politician. He was on the losing side of all the major causes he fought for and the way he fought sometimes contributed to the loss. Unilateral nuclear disarmament, Irish unification and mass nationalisation – each had his passionate and eloquent support. None happened. His diagnosis of globalisation, neoliberalism and how the wealthy and privileged entrench their power was often close to the mark, but his prescriptions scared more people than they inspired and had only a fleeting hold on the Labour party.

There was a degree of reciprocated respect between him and Margaret Thatcher as two vibrantly ideological politicians. He said of her: "She believes in something" and called her "a great teacher". The critical difference between them is that she won. On the big economic questions, Britain rejected Bennism for Thatcherism. Labour returned to power only when it embraced not Bennism, but Blairism. He regarded the man with whom he shared initials as Labour's worst leader. Yet the governments of Tony Blair did more to improve the lives of working people than Tony Benn ever managed to do.

The Labour manifesto that came closest to promulgating a Bennite Britain was the one on which it fought the 1983 election. Gerald Kaufman, an enemy of Bennery, memorably called it "the longest suicide note in history". Even more telling was Benn's response to the result of that election, which was to hail a Thatcher landslide and Labour's worst defeat since the First World War as a triumph for socialism. He told the Guardian: "For the first time since 1945, a party with an openly socialist policy has received the support of over eight and a half million people. This is a remarkable development."

That quote most perfectly captures what was ultimately hopeless about a politics that not only refused to compromise with the electorate, but wilfully refused even to think about it. If you are kind, you would say it was because he had an incurably romantic view of what the British people would vote for. If you are less kind about it, it was evidence of the impossibilist delusion that gripped far too many on the left when he was their champion. The large, historical question about him is the extent to which he bears personal responsibility for Labour's 18 years in the wilderness between 1979 and 1997. He was not solely culpable for the frenzy of fratricidal self-destruction that consumed Labour in the early 80s, but he played the greatest part in fomenting it. For a nice man, he kept some nasty company. He never disowned, and stood in the way of dealing with, the Militant Tendency and other far-left entryists who were poisoning Labour from within. He does not bear sole responsibility for the split that led to the formation of the SDP, but he was one of the most crucial triggers for it.

He always insisted that "issues"– or "ishoos" as he pronounced it – mattered more than "pershonalities". Yet it was his personality that people fell in love with in his later life. His delight in political ideas, history and debate enthralled the large audiences he attracted at one-man shows and literary festivals. He warmed up any room that he addressed. There is genuine affection in the many eulogies to him, even, perhaps especially, those from Tories who can regard him fondly now because in the end he never did pose a serious threat to the over-privileged and over-powerful. When David Cameron and Boris Johnson wax warmly, it is a measure of respect for a fascinating political figure, but also the seal on his failure.