This piece is about legacy, and about responsibility, in politics and in sport. It is the story of two great institutions, seemingly unstoppable a short time back, now stuttering and struggling, and leaving the public bemused; the story of two strong, charismatic leaders whose names became synonymous with what their organisations stood for, and whose leaderships became bywords for winning and success – Tony Blair, who left to a standing ovation in the Commons, Alex Ferguson who left with yet another Premier League medal round his neck. Since those figures departed, Labour has lost two elections under two different leaders, and is struggling under a third; Manchester United have failed to win anything under two new managers, have been knocked out early from this season’s Champions League, and their fans are complaining about the turgid way they’re playing. (Even Usain Bolt, a devoted fan, has joined in, stating that he wouldn’t want to be coached by Van Gaal.)

During the Blair years, football and politics became increasingly aligned. Tony and Gordon Brown were both keen football supporters. Peter Mandelson, fair to say, less so. I took him to two matches. The first was Marseille against Bordeaux, when we were on holiday in France in 1990; he wasn’t much interested in the football, but he was fascinated by the energy of the crowd – with 50,000 people on their feet cheering to the sound of Van Halen’s Jump as the players emerged onto the field. “Why can’t politics move people like this?” he mused.

At his second game a few years later, Hartlepool against my team, Burnley, the crowd was smaller, Peter was by then the local MP, Tony Blair was MP for Sedgefield and I was still a journalist. With our sons, Tony and I stood in the Burnley end, while Peter watched from the directors’ area of the home stand. At half time he saw fit to walk around the perimeter of the pitch to see us, a huge knitted Hartlepool scarf thrown Oxbridge-style over his shoulder, prompting the Burnley fans to sing: “Who’s the wanker in the scarf?” Burnley lost 4-1 and had two players sent off.

It was no doubt in part my passion for sport that made me so keen on the New Labour-New Football alignment, but the real backdrop for it was the growth of the Premier League as a global media phenomenon, and the pre-eminence of Manchester United, led by their Labour-supporting manager, then plain Mr Alex Ferguson.

We first met at a match in the early 90s, and a few years later, before Tony became prime minister, I interviewed Ferguson for Labour Weekly, something of a scoop as it was the first time he had really opened up in print about his politics, revealing a still-burning anger about the state of the Glasgow hospital – in his view, run down by the Thatcher government – in which his mother Elizabeth died from lung cancer in 1986. His desire to see Labour back in power seemed to be almost as visceral as my own.

We stayed in touch and became friends. Later, Ferguson began to do events, photocalls, endorsements and ghosted articles. He began to offer advice on handling pressure and mental fitness, so I also arranged for occasional meetings between him and Tony. “This is the room where I signed Eric Cantona and Andy Cole,” he told us at one meeting in a Manchester hotel room, going on to advise on dealing with big characters and big egos. “Keep them in the same room.” Players didn’t all need to get on with each other, but they needed to know the plan, and their role, and they needed to know who was boss.

As the celebrity culture developed, it was no bad thing to have the country’s foremost football manager proudly affirming his support for Labour. I could even forgive him for getting my eldest child Rory to defect from Burnley to Manchester United, not least by taking him, aged seven, into a dressing room before a match at Newcastle, and introducing him to a galaxy of international stars: Beckham and Cantona, Giggs and Scholes, Keane and Schmeichel; or walking into Rory’s bedroom one night – while visiting London for one of the famous “Cool Britannia” (copyright Time magazine) events at Downing Street – with two tickets to a Champions League game as a birthday present.

The synergy between Labour – the dominant political force – and United – the dominant football force – felt particularly strong on the night in Barcelona in 1999 that Rory and I sat with Ferguson’s sons and watched United add the Champions League to that season’s Premier League title and FA Cup.

The cabinet secretary, Sir Richard Wilson, had asked me to sound out Ferguson – if the treble was secured – and ask whether he would accept being rushed into the next honours list. He would, and did. A very New Labour honour, and he is now recognised as “Sralex” all over the world.

Alastair Campbell, Tony Blair and Alex Ferguson before New Labour had won a general election. Photograph: Supplied by Alastair Campbell

Others in our midst, such as chief of staff Jonathan Powell, were less keen on this association with what was becoming a multi-billion pound industry. The flood of money, the growing power of television and agents, and the influx of foreign owners did indeed change the nature of football. At the top level, the game was now populated by the kind of people Peter Mandelson perhaps had in mind when he said New Labour was “intensely relaxed” about people getting filthy rich, provided – the part that never gets quoted – “they pay their taxes”.

I still loved the game, though I was certainly among those who found the clashing armies of oligarch-owned multinational squads less lovable than the English clubs of old. However, with Tony winning the first landslide by projecting an exciting, confident vision of the future, I still believed that England having the most popular league in the world’s most popular sport was an important part of the New Labour-New Britain backdrop.

That New Labour was a winning machine is beyond dispute. Three general elections under Blair, three wins: the first two were enormous majorities, and even the third one – after the Iraq war – would have seemed like a great victory back in 1997. Likewise Manchester United became a byword for success, and when Sir Alex finally retired in 2013, he did so after winning the Premier League, his 38th trophy in 27 years.

So there are plenty of points to justify the alignment, plenty of points of connection and good memories – to this day, I view Tony’s head tennis with former England manager Kevin Keegan in 1995 as one of our best photocalls! But the older I get, and having spent much of the last couple of years researching, writing and talking about winners, the more I understand that there are two crucial elements of winning that are often overlooked: succession and legacy. And it is on these fronts that both Blair’s New Labour and Ferguson’s Manchester United have failed to match their other achievements.

Kevin Keegan and Tony Blair play head tennis during the Labour party conference in Brighton in 1995. Photograph: PA/PA Archive/Press Association Images

Part of the soundtrack to recent games at Old Trafford, from fans and commentators, is: “This is not the United we know.” The same could be said of Labour. It is not difficult to make criticisms of Jeremy Corbyn, and plenty of senior New Labour figures – Tony and Peter among them – have made them. I have privately agreed with much that they have said, but held back publicly on the basis that Corbyn had won the leadership. I thought he deserved some time to show that the strategy that won him the leadership could be developed into a strategy to win over the country.

After four months, however, it does feel that what was once a winning machine is being turned into a losing one; that the passions of the leader and his team are less aimed at winning support in the country than winning control of the levers of power in the party, that little is being done to set out a domestic policy agenda that will be understood and appreciated by the public, and that on foreign policy, the goal is to turn a great party of government into the political wing of the Stop the War campaign.

But whatever criticisms we may offer, Corbyn’s rise did not take place in a vacuum. We all had a role in it. In December, Peter Hyman, an important member of New Labour’s strategy team in opposition and government, and now a headteacher, wrote a searing article for the Observer about the state of the party today. He rightly noted that New Labour had developed – and delivered much of – a policy agenda that was bolder and more ambitious than anything proposed by Corbyn or David Cameron. But he also argued that we failed to develop talent, failed to cement organisational and cultural change in the party and failed to secure our legacy.

He was right: we had three years in opposition under Tony’s leadership, then 13 years in power, the last three under Gordon. Yet just five years after Labour’s longest-ever period in office ended in 2010, the party elected as leader Jeremy Corbyn, as far from New Labour as it was possible to be.

There are Blairites who say the problem was Gordon ever becoming prime minister; Brownites who say that Tony should have gone earlier; both bemoan Gordon’s indecision on the question of whether to hold a snap election in 2007, shortly after he became prime minister; many say that we chose “the wrong Miliband” after Gordon stepped down. Whatever the factors that got us to where we are, and however much I hate the Blairite/Brownite labels that suggest division that will never end, Hyman is right to call this an existential moment for the party. And he is right to say that this is not just about Corbyn.

I’m afraid we do have to wind back the clock to 1994 and the oft-asked question of whether there was a deal between Tony and Gordon about who should stand to become the next Labour leader, following the death of John Smith. Gordon certainly thought there was – though Tony, like many top-flight politicians, is capable of leaving a colleague with the impression they might want to have, without saying the exact words they want to hear. Two things flowed from the TB-GB agreement that only Tony should stand as leader at that time: Gordon was determined he should be next; and for all the grief along the way, Tony ultimately sided with him, and helped to bring about the “smooth and orderly transition” that we eventually managed.

So when Tony left office in 2007, despite his and others’ doubts, we were down to a leadership contest of one. The consequences of that were still with us in 2015 when Corbyn beat what was widely seen as an uninspiring field. A generation of possible contenders had taken their talents elsewhere: David Miliband had left for the US; Alan Johnson was enjoying a new lease of life as a writer; John Reid, Alan Milburn, Alistair Darling, David Blunkett, Charles Clarke … plenty of people had been touted as possible leaders back in the day, but they had been seen off or moved on. It was like a football club trying to put together a new team, needing a new captain, when half of the best players had gone.

This dilemma of how to manage succession remains a pressing dilemma in politics today. I wonder about David Cameron, who is serving his second and final term as prime minister, and whether he will help or hinder George Osborne by so clearly favouring him as the next leader. Then there is Angela Merkel in Germany, and the question of whether she will try to control the appointment of her successor. Merkel is to my mind the most impressive leader in world politics today, someone with a fearsome reputation, almost Gordon Brown-like, for minimising any threat from potential rivals. But with perhaps one more mandate ahead of her, will she start to allow others to emerge so that her party and her country are able to have a proper choice of potential leaders to follow her?

Gordon Brown and Tony Blair at a general election campaign press conference in 2005. Photograph: Odd Andersen/AFP

Hyman always thought that Gordon should have faced a challenge. He felt that too many of us – Tony, me, Peter Mandelson, the strategist Philip Gould, much of the cabinet and parliamentary Labour party from John Prescott down – were cowed by Gordon into making sure we played along with his determination to ensure that he alone could be considered as Tony’s successor. There is something in that.

Indeed, when Philip Gould and I were putting together an election strategy for 2005, we went to see Tony at a hotel in Scotland and persuaded him that to have any chance of a decent win, we had to fight it on the economy. That meant that unless he was going to sack him, we needed Gordon at centre stage as chancellor, and we would only secure that if Tony made it clear that this was his last election and that he would back Gordon to replace him. Right or wrong? I don’t know. What I do know is that the imperative of winning at that time took precedence over any thinking about legacy. To make the football analogy again, going into the election without Gordon would have been like Manchester United playing a season without an in-form Wayne Rooney.

A few years later, when some thought David Miliband looked like a better bet for Labour, Gordon had enough power in the party to stay as prime minister. Peter Mandelson, who had been angry with me and Philip for conceding too much to Gordon in 2005, was now a key player in the Brown team. So when, amid a turbulent pre-election period in 2009, James Purnell resigned as culture secretary, perhaps expecting a mini-exodus that would fatally undermine Gordon and open the way for David, Peter helped to prevent it. I don’t criticise him for that. It was part of his job at the time. Also, for all our doubts, Philip and I, Labour team-players, also went back to help Gordon campaign in 2010.

Just as Tony’s leadership did not see the development of sufficient new talent, nor did Gordon’s. He did some good things in office and handled the global financial crisis well. But the bold policy agenda that we had imagined he had lined up for when he became prime minister did not materialise – and he continued to define himself against Tony rather than David Cameron. Meanwhile, Tony, determined not to repeat the mistakes of Tory prime ministers in causing grief for their successors, essentially disappeared from the UK scene, allowing a process that began with people failing to stand up for his considerable record as PM, and developed over the years into a tripartite trashing of it – by Tories, the media and by many Labour people, too.

Legacy matters for all sorts of reasons. Historical reputation matters and the Tories use it far better than we do. Why else do they bang on about “the mess we inherited from Labour”, let alone its predecessor, the “winter of discontent”? It is to their eternal benefit that Winston Churchill is seen as Britain’s greatest ever leader. It is to their advantage that they stay loyal to the memory of Margaret Thatcher, and to our disadvantage that the party has played its part in defining Tony, our most successful leader, so negatively. The Tories understand that history has potent political force in the present, and as a political party rightly obsessed with winning, they use it.

In the closing days of the 2010 campaign, in Manchester, Gordon gave one of the most rousing summaries of our record that I have ever heard. One after another, in short, punchy sentences, he listed the dozens of changes for the better that the Labour government had made. As the list grew longer, his voice grew louder, and with it the applause of supporters loving every reminder of why Labour in power is always better than the Tories in power. But it was too late. Days later, David Cameron was prime minister; and by 2015, the Tories had managed to make “the mess we inherited from Labour” the soundtrack to political debate, every bit as much as we had made it “things can only get better” in 1997. The record had gone by default, and Labour New and Old must accept responsibility for that. It is not the media’s job to remind people of the good Labour governments do. It is certainly not the Tories’ job. It is ours.

Could we have won in 2010 if, instead of spending so much energy dealing with “our friend next door”, as Tony called him, we had worked on developing other talent who might by then have become genuine leadership material? We will never know.

Tony was often tempted to sack Gordon but ultimately he “kept him in the room”. This is where politics and football are different. More than once, Alex Ferguson asked me: “Why doesn’t he just sack him if he’s causing so much grief?” But if a manager “sacks” a player, that player has to find another club. When he felt his deputy Brian Kidd had his eyes on the top job, Ferguson could move him out, and that meant Kidd moving on, to Blackburn. In politics, sacked players stay on the pitch. But I do think that back in 2007 Tony should probably have shown more obeisance to the old adage about never picking your own successor. Here too, perhaps, there is a point of connection with Manchester United.

Given all he won for the club, the Ferguson legend within the Manchester United family is as secure as his statue outside the Old Trafford stadium. But his legacy is less secure and, as with Tony and New Labour, that is in part down to what has followed him, and his role within it.

Ferguson had been in charge so long, so successfully, that it was always going to be tough to get the right successor. His longevity meant he had to some extent “become the club” in a way that, outside of dictatorships, political leaders never really “become the party”. Margaret Thatcher got close but even she, unlike Sir Alex, did not choose the timing of her own departure.

Sir Alex, with a much wider field to choose from than Tony, had nonetheless made the same mistake in picking the man who would follow him. Perhaps understandably, given Sir Alex’s status, the normally hard-headed American owners of the club, the Glazer family, allowed the outgoing manager to take ownership of his own succession. Football does funny things to business people. I doubt they would have taken the same approach in any of their non-sport businesses. Looking back, the Glazers must be starting to wonder, whatever the flak they might have taken at the time, whether they should have made Ferguson’s departure genuinely the end of an era, and had proper succession planning in place that was not dependent on the man who was leaving.

Just as Tony, however reluctantly, helped ensure that Gordon followed him, so Sir Alex, much less reluctantly, handpicked his successor, David Moyes, inviting him to his Wilmslow home and telling him that the biggest job in the English game was his, later making a passionate appeal in a packed stadium to the fans to get behind Moyes in good times and bad. When it became apparent that Moyes’s appointment was failing with players and fans, the board, which includes Sir Alex, withdrew their support and along came Louis van Gaal.

Alex Ferguson and his eventual successor David Moyes, in 2007. Photograph: John Sibley/Action Images

With Moyes’s Dutch replacement too, however, the Ferguson imprimatur was needed. The fans needed to know, once more, that he approved. The appeal of the first appointment appeared to be in its echo of a younger Fergie moving down from Aberdeen – Moyes, too, was Scottish, working-class, passionate, with a formidable work ethic and experience with a well-run club, Everton. Van Gaal, meanwhile, was an echo of the older Fergie – his own man, vastly experienced, capable of dealing with big names, trophies galore under his belt, tough with the media. But the longer this season goes on, the clearer it becomes that, despite United’s win at Liverpool at the weekend, Plan B is not really working either, and there may soon have to be a Plan C.

It is not unusual for people leaving a job to want to handpick a successor, stay involved and be viewed in hindsight as irreplaceable. (Think of Thatcher and Major in politics, for example.) And, of course, as the men in charge, Moyes and Van Gaal must take responsibility for results. Van Gaal in particular has spent a fortune on new players and Sir Alex is no longer the one overseeing training or devising and explaining tactics. Yet much of the spine of the team remains from the Ferguson era, and on the bench alongside Van Gaal is Ryan Giggs, as strong a connection to the Ferguson era as could be imagined.

Perhaps the problem has been that Sir Alex’s post-managerial strategy was designed as if the good times were bound to keep rolling. There was no real resetting of expectations against the fact that such an asset as Ferguson, with his experience and winning mindset, was being lost.

One of the first things Ferguson did after stepping down was to publish his second autobiography, and then a book on leadership. Inevitably, he has been drawn into giving public commentaries on the club he still serves. His observation that he had only managed four world-class players during his tenure – all of whom had retired – was not exactly a morale-boosting expression of support for the current set-up. Instead, it seemed to suggest that success had been more down to him than to the hundreds of players he coached.

It is virtually impossible for the new manager to match up to the old one in this case. Sir Alex and Arsène Wenger at Arsenal are probably the last two top club managers who will be given time, and the occasional long barren runs, to build success for the long term. Moyes was given just a few months and, as results began to go against him, the increasingly isolated looking manager started to feel very negatively the presence of a glum-looking predecessor in the directors’ box behind him – an easy cutaway for the TV producers. I saw Moyes in his office in the week before he was sacked. He cut a lonely figure, clutching at the straw that the club could not be about to sack him, as reported in the media, because no one had said anything to his face to suggest they were unhappy with his work. Days later, the banner behind the goal, proclaiming Moyes as “the chosen one” was removed as quickly as it had been put up. That showed a ruthlessness that was perhaps inevitable, and more in line with the Tories’ approach to struggling leaders, than to Labour’s.

So now we have Jeremy Corbyn, as surprised at being leader as everyone else is. But this did not happen in a vacuum. Tony begat Gordon begat Ed begat Jeremy.

I know that if Labour becomes an anti-American, anti-Nato, weak on defence, soft on terror, unilaterally disarming party of protest, along the lines that Corbyn appears to want to take it, then David Cameron’s successor can expect a long haul in office. But, although I can and will defend Tony over Iraq, I also know that we cannot overlook the fact that widespread opposition to the war, and anger about the aftermath, played a big part in Corbyn’s rise.

I will also continue to defend the professionalism we brought to our strategy and communications, take pride in the fact that political organisations around the world have adopted much of the New Labour approach, and lament what passes for a communications operation now. But, again, it is true that we must accept that Corbyn won in part because we allowed our approach to communications – which was simply a necessary shift towards a more disciplined approach – to become defined as spin.

Tony begat Gordon begat Ed begat Jeremy: Tony Blair and Jeremy Corbyn at the service of remembrance at the Cenotaph in November. Photograph: Facundo Arrizabalaga/EPA

When Gordon said “no more spin, no more sleaze”, he was playing into an attack on New Labour that we should have rebutted better. When Ed Miliband spoke endlessly about “learning lessons” from the New Labour era, he too played into, rather than challenged, a negative image of Labour that ultimately benefited David Cameron more than it did himself. I am less angry about the changes to the electoral system, which allowed people who paid £3 to register as supporters the chance to vote and ultimately helped Corbyn win so emphatically – indeed Labour as a mass-membership party was one part of the New Labour vision we certainly didn’t manage to deliver – than I am about the step-by-step distancing from a winning approach to politics and campaigns. Gordon moved a little away from New Labour, and lost. Ed moved much further away, and lost badly. And now we have Corbyn.

I didn’t vote for him; indeed I may have coined the ABC – Anyone But Corbyn – line of attack during the summer. But the contest certainly showed up the dearth of talent. The figures of old had gone, and a number of current MPs, including Chuka Umunna and Dan Jarvis, dipped toes in the water but decided the time was not right to dive in. The debate among the eventual ABC field of Andy Burnham, Yvette Cooper and Liz Kendall was uninspiring, their messages lacked clarity and their campaign methods felt tired. By now the New Labour brand and political playbook were virtually toxic. By contrast, Corbyn was able to mount one of the most remarkable campaign wins any of us can recall. Even if Leicester City win the Premier League this season, that would not match the scale of what Corbyn managed to achieve. This was a man whose initial objective was merely to get on the ballot – and he almost didn’t achieve that – who raced just weeks later to a huge win.

I do not dismiss all his supporters as Bennites, let alone pseudo-Trots (though there are some of those), and I think many more come from the party’s Kinnockite soft left, have a newfound interest in politics, and really would like Labour to win. But as I know from any Labour gathering, we now seem to spend all our time talking about the party and not enough about the Tories and the public. The gap between party and public, the vast bulk of whom are not inside the political bubble, is widening. Virtually every day, people outside the political bubble, who now feel politically homeless, stop me to ask what the hell is going on.

Planning succession and legacy while doing the job of prime minister is not easy. The present moment is always so pressing that it is difficult for a leader to give much priority to what comes after him. But, in truth, both under Tony and Gordon, we should have done more, and done better.

In the fortnight before Christmas I spoke to audiences totalling several thousand people – from schools and universities to charities and businesses. Inevitably, Labour’s situation came up again and again. Can Corbyn win? Will he last? Did we pick the wrong Miliband? Will David come back? Who will the next leader be?

As to whether David will come back, I doubt it. As to whether the current leader can win, I throw that to the audience and ask them not who they want to win the next election but who they think can win it. Both George Osborne and Boris Johnson get a lot of raised hands, any other Tory contender next to none. I had 11 hands raised for Jeremy Corbyn becoming prime minister, seven in a school, three at a business dinner and one at a college.

So somehow we seem to have ended up with a leader who is unelectable but unassailable. Unelectable in the eyes of the non-political public, unassailable because he is exactly what those inside the bubble want: something different. For some in Momentum, the group that was set up to turn Corbyn’s support from the leadership election into a broader political movement, the hope is that his unassailability as leader might be enough to halt any return to the centre ground. But that is not enough for anyone who wants Labour back in power, and who knows that though times have changed, being sound on the economy, strong on defence, and fairer than the Tories remain key messages in a winning formula.

I loathe this government and many of the things it does. But for the first time in my life I cannot really answer with any confidence the questions “What do you think will happen to the Labour leader and the party? And what do you think we should do?” When Ed was elected I said privately – though unfortunately it leaked out – my worry was that he would make the party feel OK about losing but wouldn’t win. I was right about losing but wrong about anyone, other than the Tories, feeling good about it.

Losing didn’t feel good on 7 May and it feels even worse now as the Tories make the rich richer and the poor poorer, preside over an explosion in rough sleeping, herald the return of winter NHS crises that New Labour ended, bring about a housing crisis and social cleansing from gentrified areas, screw up our relations with Europe, exploit Labour’s weakness north and south of the border to tighten their grip on the levers of power in order to make sure that their narrow, top-down, elitist vision for Britain remains dominant. Though Labour must do a better job both of opposing and of putting over an alternative, we cannot just blame Corbyn for that; the reality is that we all played a part in getting us to where we are.

As for David Cameron and Angela Merkel in politics, or Arsène Wenger in football, perhaps the lesson is that if they want to secure their legacies when they go, they should not interfere too much in the process to decide who succeeds them.

• Alastair Campbell’s bestselling book Winners And How They Succeed, includes interviews and profiles of Blair, Merkel, Ferguson, Wenger and Mourinho.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.