Inevitably,​ the first thing I did when I got my copy of the one-volume edition of The Benn Diaries (Hutchinson, £30) was to look up Jeremy Corbyn in the index. He appears about as often as you’d expect, 15 times in total, scattered at regular intervals across 24 years, from 1983 until 2007. (The diaries end in 2009, five years before Benn’s death.) Corbyn rarely shows up on his own; more often he features in long lists of names from the left of the Labour Party who have gathered for rallies, protests and funerals. He is there at an anti-poll tax demonstration; at a Sinn Féin press conference; at a meeting of the Palestine Solidarity Campaign; at the laying to rest of Tony Banks. Some of these entries are inadvertently comical. Writing of Ralph Miliband’s funeral at Golders Green Crematorium in 1994, Benn notes: ‘Anyone on the real left of any significance was there. Jeremy Corbyn couldn’t make it.’ For the most part, the mentions of Corbyn are respectful, if a little detached. He is a good man to have alongside you on the platform or to put up in front of a sympathetic crowd. You know where you stand with Jeremy. You just don’t often notice him.

The one occasion when Corbyn takes more of an active role comes in 1988 when he chaired the meeting of the Campaign Group that chose Benn as its candidate to challenge Neil Kinnock for the party leadership, despite the opposition of some members of the group, including Margaret Beckett and Chris Mullin. Benn describes himself as ‘peaceful in my own mind’ about this outcome because it had been a collective decision, not an act of personal vanity. Corbyn’s chairmanship of the group helped salve his conscience. Benn was less at ease when the contest took place in the autumn and he found himself humiliated, winning just 11 per cent of the electoral college, thereby cementing Kinnock in the post. He felt let down by the unions, which had encouraged him to stand and then failed to support him. This sequence of events appears to have confirmed Benn in his suspicion that the goal of capturing the party leadership was a kind of trap for the left. It often appeared within reach, yet it had a habit of coming back to bite them.

The idea that it would be Corbyn himself who finally won the prize is nowhere anticipated in these diaries. But the temptation to try again is a constant theme, and it is not just Benn himself who is drawn to the flame. Eric Heffer, Audrey Wise, Michael Meacher, Ken Livingstone and others feature in earnest discussions about whether the time is right for another attempt to capture the flag and how the forces are assembled. The votes of the unions and the membership seem to allow the possibility that the parliamentary party can be outflanked, if only they can be trusted to deliver. Some of Benn’s misgivings about this pursuit relate to his puritanical streak and his suspicion that personal conceit is the great enemy of principled politics. He regularly tests himself for the besetting sin of prideful ambition. When the subject of the leadership comes up he usually tells his diary that it is a dangerous distraction. He also feels it is perilous for the causes he believes in. When Livingstone floats the idea of standing in 1992 following the departure of Kinnock, Benn writes: ‘I think on the whole I’d prefer a right-winger who had to work with the left than a so-called left-winger who had to prove his responsibility by swinging to the right.’

After Michael Foot won the leadership in 1980, Benn comments on the supposed ‘magic power’ that a new leader of the party acquires. ‘Anyone who becomes Labour leader becomes a little bit different … With a new suit and a haircut, Michael already looked a bit different.’ This is partly sour grapes: Benn was far from immune to personal vanity, hence the need for regular self-scourging. But it also reflected his strong sense that the role was a poisoned chalice for anyone seeking to hold fast to his or her convictions. A new suit and a haircut will never be enough to persuade the doubters – and the idea that Foot had undergone a makeover now seems rather quaint – but it is nonetheless a step on the slippery slope to presentational politics. The contest that made Benn happiest was his full-throated assault on the deputy leadership the following year, when he came within a whisker of defeating Denis Healey. ‘It has been a staggering result,’ he wrote in his diary in the immediate afterglow of heroic defeat, ‘with all the media against us, the most violent attacks by the shadow cabinet, the full intervention of Michael, the abstention of a group of Tribune Group MPs’. That Kinnock was one of those MPs allowed Benn to nurse a grievance that sustained him for the next decade or more. To lose by just 0.8 per cent of the total vote had been, as he put it, ‘the best possible result, because if I had won by 0.8 per cent people would have shouted “cheat”.’ Far better for Healey to ‘hold the post but not have the authority’. The great danger of winning was that this would become true for his side instead.

One of the causes that Benn consistently believed should trump the siren call of high office was Europe. Here he sometimes found himself out of step with his comrades on the left, including Corbyn. In 1992 he was passionately committed to a referendum on the Maastricht Treaty, on the grounds that Parliament was abrogating the sovereignty of the people. Not only was the entire Labour shadow cabinet under John Smith opposed to such a view, so too was a group that included Corbyn, Dennis Skinner and Bernie Grant. ‘It disoriented me a bit,’ Benn writes, ‘because you don’t like to go against your own people.’ Still, Benn felt he had no choice but to press on, and was shot down in flames at a meeting of the PLP. It wasn’t only his friends on the left who let him down. In July that year he recounts hearing Mrs Thatcher, newly ennobled, make an attack on Maastricht from the House of Lords. ‘A packed House of Lords listened to her in absolute silence – nobody agrees with her in the House of Lords, and anyway by accepting a peerage she’s degutted herself. Addressing a message about democracy from there is totally incredible.’

The turmoil inside the Conservative Party brought about by Maastricht – culminating in Major’s futile attempt to see off the ‘bastards’ in his cabinet – helped confirm the view that the issue of Europe was potentially fatal for the Tories. Benn’s diaries suggest a different story. The party most at risk of being destroyed by Europe was Labour. The Tories could always refract their differences through the cut and thrust of high politics, whereas for Labour there was the real risk that any divisions would prove insurmountable. Benn was never reconciled to the idea that social democracy in Britain could be achieved if decision-making power was franchised out to the bureaucrats in Brussels. Yet from the late 1980s on almost the entire parliamentary party took the opposite view, coming to believe that the EU represented the best bet for enshrining workers’ rights. In Benn’s eyes, this amounted to giving up before you’d even started. He also thought it risked stoking nationalism around Europe when the workers came to realise how they’d been betrayed. His diaries echo with dark warnings about the relentless drive towards greater European integration and what it would do to democracy and to Labour. In October 2004 he listens on the radio as Blair tells Labour’s MEPs they have to support the new European Commission. He despairs. ‘They are just privates in General Blair’s army. The principle of representation has died in the Labour Party.’

Now here we are in 2017 and the two things of which Benn did not quite dare to dream – a genuine leftist at the helm of the party and the UK on the verge of quitting the EU – have both come to pass. The Labour Party lies broken as a result. I don’t want to claim that Benn foresaw this. But his diaries give the sense that he had an instinctive grasp of some of the deeper forces at work and he knew all good socialists need to be careful what they wish for. Just as the leadership was a trap for the left, so too was Europe, which risked leaving Labour without any principles to fall back on. Benn understood what he wanted to happen, but he also understood that he was most comfortable pushing for these things at some distance from actually achieving them. Had he lived to see them achieved, he would doubtless feel vindicated. But he might also feel that his life’s work was at risk of being undone.