In the summer of 2000 I was commissioned by Harper’s Bazaar magazine to write about the young gilded special advisers who were working for Tony Blair and Gordon Brown or orbiting around them, or who were close to Peter Mandelson. Most of them wanted to be MPs. I did not know them personally but I knew a lot about them – about how they lived, worked and socialised. Some of them lived together – indeed, even slept together. They were intelligent: all had been educated at Oxford or Cambridge, and some had known each other from student days. They were well connected, competitive, football-loving metropolitans, liberal, good Europeans. They were fascinated by US politics (some of them had done graduate research at American universities or worked on campaigns for the Democrats). They’d studied closely how Bill Clinton and his advisers had remade the Democrats – through trian­gulation, message discipline, media mastery – as a centrist, optimistic, pro-capitalist, election-winning force.

Older Labour MPs naturally resented these “Young Turks”. More, they envied them. John Prescott called them “teeny boppers”; others called them “faceless wonders” and much worse. They were considered cliquey, superior in manner, even conspiratorial – a bit like the classics students in Donna Tartt’s Secret History, who are punished ultimately for believing in the myth of their own intellectual superiority. Blair and Brown worked them hard but believed in them, just as they believed in themselves.

It was obvious that the newcomers thought that one day they would be running not only the party but the country. They had a sense of purpose and mission as well as self-righteousness. They were the best and the brightest of their political generation and comparisons were made between them and the young policymakers who had worked for John F Kennedy and then Lyndon Johnson in the 1960s, and whose misjudgements took America into the Vietnam War. A path was being cleared for them – and they were heading in only one direction, to the summit.

Their names are now very familiar: David Miliband, Ed Miliband, Ed Balls, James Purnell, Andy Burnham. There were others among the group who are less familiar, such as Peter Hyman and Liz Lloyd, who both worked in the Downing Street Policy Unit. (Hyman later became a Newsnight pundit and set up a successful free school.)

Balls and David Miliband were considered the best of the group, and Balls, in particular, was admired at the Treasury for his intellectual powers.

“Ed was really exceptional,” a former politician who knew him back then told me. “In terms of managing the Treasury he was the most brilliant sort of aide-de-camp. That’s not a pejorative expression: I mean Gordon needed that sort of intellectual grip and energy and the thuggishness as well – you know, to turn the Treasury into an instrument for Gordon. Probably only Nigel Lawson in our lifetime had a similar mastery of that department. And Ed was absolutely fundamental to that in all kinds of ways. He’s not a great visionary and thinker, but he’s got huge intellectual grip.”

Back in 2000 I called this group Labour’s Golden Generation, and Harper’s, in the photoshopped illustration accompanying my piece, brought them all together in a mock football line-up. We dressed them in the shirts of the Demon Eyes football team, for which some of them played and which was named after the notorious Conservative 1997 election campaign poster. Conceived by M&C Saatchi, the poster depicted a demonic Tony Blair, his burning red eyes staring out from a black background, above the slogan “New Labour, New Danger”.

Although I did not write about them for Harper’s, because they were already MPs, Yvette Cooper and Douglas Alexander were part of the Golden Generation as well. Cooper was only 28 when she won the safe seat of Pontefract and Castleford at the 1997 general election. She married Balls the following year, having met him when they shared an office as young advisers.

Alexander became an MP in November 1997, at a by-election. He was 30. Together with Jim Murphy, Alexander (who before entering parliament had worked for Brown as a speechwriter) ran David Miliband’s leadership campaign in 2010. Neither would ever again be trusted by Ed Miliband, who won that contest, which pitted the Golden Generation against one another. Burnham was also a contender in 2010 and so was Balls.

There were others, in and around the core group, who burned brightly and burned out (Derek Draper), or who peeled off early to work in PR, consulting or television (Benjamin Wegg-Prosser, Tim Allan), or who never sought elected office (Liz Lloyd, Mark Leonard, Spencer Livermore, who was a long-time Brown favourite).

“The emergence of these bright young things is largely down to Tony Blair,” Derek Draper, a former adviser to Mandelson who became a lobbyist and then a psychotherapist, once told me. “He was the first Labour leader to pluck young Oxbridge graduates out of nowhere and put them in charge of his office. This put people’s backs up. It even put my back up, and I was one of the young insiders myself. ‘Where have they been in the party?’ I kept asking.”

Naturally, some in the group were in the orbit of Brown at the Treasury (Balls, Ed Miliband) and some were in Blair’s (David Miliband, Purnell). Blair and Brown both became MPs in 1983 and, in the aftermath of the Labour split two years earlier, they experienced the party’s devastating loss to Margaret Thatcher under the leadership of Michael Foot. So Blair and Brown were formed by these desperate times and they felt that there was no option but to combine tactical ruthlessness with strategic vision as they set about remaking the party, dragging it across the hardest ground to where they believed it had to be. The death of John Smith in May 1994 accelerated the process but it would have happened anyway.

Blair and Brown had different protégés but they shared a sense of mission. “We did it,” they must have thought many times as things did indeed begin to get better: “we created New Labour.”

They picked out people – clever, presentable, driven young people – who they believed would carry on their legacy when the time came. Along the way, older, gifted politicians – such as Alan Milburn, who was a difficult man but had an interesting personal story, or Estelle Morris, who was a good minister but suffered a loss of confidence – were squeezed out or gave up altogether.

We were witnessing the beginnings of a party being hollowed out and of a deepening disconnection between Labour MPs and their core supporters, which, in time, would empower the SNP in Scotland and create an opportunity for Ukip in England.

“I can give you a whole cadre of these people who weren’t the Oxbridge elite, the special advisers and all of the rest of it,” one former MP told me, “but they were politicians and they did have a sense of what voters wanted and they had a way of communicating with voters that these guys [the young MPs and special advisers] never did. Just never did. And as a result, it was a profound misunderstanding of what democratic politics was about. It’s not a seminar.”

Call it vanity, call it hubris, but whatever you call it, the foundations of the mansion built by Blair and Brown were neither deep nor strong. When pressure came, it began to crumble and then, after the 2015 election defeat, it collapsed, and now Jeremy Corbyn and his followers are roaming through the ruins, creating a new party of anti-capitalist renegades.

***

I’m not sure quite what divided the Blairites and Brownites politically, in retrospect. Perhaps the Brownites were more Eurosceptic, more numerate and economically literate, and were always resolute in their opposition to the euro. The Blairites were more internationalist, perhaps (though Brown was a convinced Atlanticist), liberal cosmopolitans who were idealists in matters of foreign policy. Blair’s April 1999 Chicago speech remains one of the defining texts of liberal interventionism. Justifying the Nato intervention against Serbian aggression in Kosovo, he defended Western “values” and made the “moral” case for getting “actively involved in other people’s conflicts”, which meant intervening militarily against despotism where necessary. (When David Miliband ran for the leadership of the party in 2010, he defended the Iraq War. His brother, who was not an MP at the time of the invasion and was at Harvard, claimed to have been against it from the beginning.)

But, in essence, though there was much personal rivalry between the two camps, they were united in believing in the New Labour project. “Fundamentally there wasn’t a big ideological gulf between Blair and Brown,” Ed Balls says now. “There wasn’t even, really, a big policy divide – in 1997, Tony was the Eurosceptic rather than Gordon. Things changed in terms of how he saw himself as a leader, but I don’t think he ever really wanted to join the single currency. He just really wanted to look like he wanted to join the single currency because he thought that that was important for Britain, and [for] his standing as PM at that time.”

When the main opposition party is so weak, as Labour still is today, Balls says, “the prism of politics becomes the succession within the governing party”, hence the obsession with the rivalry between Blairites and Brownites.

Balls and I spoke last Saturday as he made his way to watch Norwich City, where he is chairman of the club.

“When it came to the crunch, Tony so often backed Gordon that it used to drive him mad,” he told me. “There was a not ideological divide on the private sector in the health service, or markets and the health service. But what happened was ‘Gordon is anti-reform’ became a rallying cry for those who were trying to prevent him defeating Tony. So I think it was much more of a political thing, rather than about policy and ideology. And I think, if you go back to those debates in that period, it was the generation before us. So when you pick that ‘Golden Generation’, as you call them – me, Andy, Ed, David – none of us were really having that fight in the early 2000s, and none of us were having that fight between ourselves, certainly. The struggle was [between] the slightly older generation: some combination of Alan Milburn, Stephen Byers, Charles Clarke, who thought they had an opportunity to succeed Tony rather than Gordon. The Blair-Brown divide was never really a divide among our generation at all.”

Marginal differences, then. Certainly none of the Golden Generation doubted that the left had been vanquished. The parliamentary Bennites in the Socialist Campaign Group – Jeremy Corbyn, John McDonnell, Michael Meacher, Dennis Skinner and the rest – were considered cranks. They offered no threat.

Moreover, the Conservatives were in retreat, decisively beaten in 1997, as they would be again under the leadership of William Hague in 2001. Blair and Brown were contemptuous of the Tories, even when David Cameron and George Osborne took control of the party. They simply didn’t take them seriously, because they considered them beneath taking seriously.

The Golden Generation, along with Blair and Brown, believed, too, that devolution would settle the Scottish national question. Devolution would “kill Scottish nationalism stone dead”, boasted the future Labour defence secretary George Robertson in 1995. In the 1997 general election Labour won 56 of the 72 seats in Scotland; the Tories won none. Scotland belonged to Labour. (Today both parties have only one MP each in Scotland; the SNP has 54 of 59.)

Even as late as 2011, on the eve of the Holyrood election that the SNP won in a landslide, Ed Miliband was blithely complacent about what was happening in Scotland. As a quasi-Marxist, he never really understood nationalism or culture; he understood only political economy. By the time of the 2014 independence referendum, he was like a fugitive being hurried in and out of Scotland, never daring to stay for long. Yet it was on his watch that the party finally lost Scotland, even if the death of Labour Scotland was decades in the making.

***

So, in 2000, everything seemed set fair for the Golden Generation. Nothing could stop them from dominating public life for decades to come. Apart from one another, as it turned out.

As we know, they never quite made it, not in the way they would have wished, even if some of them had ministerial careers. The painful truth was that before any one of them reached the age of 50, it was too late, already too late: the best of their political careers was in the past.

Consider where they are today.

Ed Balls, who longed to be chancellor and ran for the leadership in 2010, has left politics after losing his seat at the 2015 election and is now cavorting on Strictly Come Dancing. The book he has just published, Speaking Out, is an unexpectedly engaging read, accessible, warm and candid. He writes nostalgically about his early years in the Treasury working closely with Brown. That was his time – and he was not yet 30.

James Purnell is expected to become the next head of BBC Radio, having quit politics in despair in June 2009. On abruptly resigning as work and pensions secretary, he called for Brown “to stand aside to give Labour a fighting chance of winning the next election”. He hoped, perhaps believed, his resignation would lead to a rebellion against Brown and that David Miliband would become prime minister. But no one followed Purnell over the top and he was shot down in no-man’s-land. When I met him in Soho for coffee not long after his resignation, he seemed liberated. “The way we do politics in this country is infantile,” he said.

David Miliband, who resigned from parliament in 2013, is based in New York doing important work as the head of the International Rescue Committee, having had his political career destroyed by his brother. There is some hope on the Labour benches that he might one day return to the Commons. “Had Ed not stood against him, none of this would have happened,” a senior Labour figure said to me, referring to the 2015 defeat, the capture of the party by the radical left and Brexit. It is difficult to see a way back for David, in the present circumstances, should he even wish to return.

Andy Burnham, having twice stood to be leader of the party, is preparing to leave Westminster as he seeks to become the first mayor of Greater Manchester. This is surely recognition that he feels Labour has no chance of returning to government any time soon. Burnham’s second run for the leadership in 2015, when, despite being the early front-runner and favourite, he was defeated by the 100-1 outsider Jeremy Corbyn, was a sad failure. He posed as the anti-Westminster candidate, the People’s Andy, the boy who lived outside the “Westminster bubble”, but he did not seem to know his own mind or what he wanted for the country. By the end of the campaign his pledges and promises were received with derision.

Douglas Alexander, who was shadow foreign secretary under Ed Miliband and Labour’s 2015 election co-ordinator, was perhaps the most obsessive US politics watcher of all that generation. He was credited with bringing David Axelrod, the American political strategist, to work on Labour’s 2015 campaign. Articulate and personable, he humiliatingly lost his Paisley and Renfrewshire South seat to a 20-year-old student, Mhairi Black of the Scottish National Party, who has since become something of a social media star. Alexander has said very little about Labour’s defeat and its collapse in Scotland, and now advises Bono of U2.

Ed Miliband and Yvette Cooper, who ran for the leadership in 2015, are still MPs, of course, but each in a different way is struggling to recover from defeat and to find purposeful self-definition. Miliband is now deeply resented in the party. Not only did he lead Labour to an abject election defeat in propitious circumstances, he introduced the new rules by which the party elects its leader, opening the way for the Corbyn insurgency. Many of those who served in the Blair and Brown cabinets resent Miliband’s reluctance to defend the record of those governments. “I thought there had been enough of an opening up of debate under Ed [Miliband] that if Jeremy was on the ballot paper we would do quite well,” Jon Lansman, a close adviser of the Labour leader and chair of the pro-Corbyn Momentum group, told the New Statesman recently. The left has mocked Miliband’s recent demand that Corbyn resign.

***

I put it to Ed Balls that his generation failed. How else to account for the rise of Corbyn and the collapse in support for moderate social democracy among Labour members and activists?

“It might be that every generation has their time, and as it happened for our generation, it has happened to Cameron and Osborne,” he said by way of a reply. “And it may be that politics has become more – well, your shelf life goes down. It’s harder to regenerate politically, when you think of how long people like [Harold] Wilson or Denis Healey managed to be around. Maybe that’s harder these days. It may be that we made a mistake, and we should talk about that. It may be that, because we were people who succeeded early, and therefore became identified with the mainstream, we became casualties – on both sides of politics – when the centre ground gets rejected for the extremes . . . I don’t think we ever got complacent. I think once we lost in 2010 there was no sense of complacency.”

He found the 2010 defeat especially difficult: after all, he had known only success. “We’d been in government for 13 years . . . But when you go out of government in those circumstances, it feels like something that’s not going to be short-lived or temporary. There are some people who wait all their lives to get to a senior position in the civil service, or in government, or get into the cabinet, and they get it in their sixties. But because it happened for me on my 30th birthday, and the cabinet in my early forties, maybe you have a sense that your time may have been at an earlier stage. I definitely felt that in 2010.”

One recent morning I had coffee with Eddie Morgan, who has worked for the BBC and ITV as an editor and producer. He was close to Cooper and Purnell in the late 1980s when he read philosophy, politics and economics at Balliol College, Oxford (where they overlapped with Boris Johnson) and he worked alongside the Golden Generation in the early 2000s when he was Labour assistant general secretary. “How foolish that now all looks,” he said, reflecting on the Blair-Brown conflict – the so-called TBGBs. “Talk about the narcissism of small differences! They really were a golden generation, weren’t they? Yet even back then I was struck by how uncollegiate it was. Everyone had their baronial fiefdoms. There was not a lot of glue between them.”

This might explain why no one emerged as the leader of the pack, as David Cameron did among the Tory modernisers, and four of them – Balls, Burnham and the two Milibands – took each other on in the 2010 leadership contest.

For Morgan and his friends, working for the Labour Party offered thrilling possibilities. “After the ERM debacle [Britain was humiliatingly forced to withdraw the pound sterling from the European Exchange Rate Mechanism in September 1992], the Tories were on the way out. They’d been in power too long. The feeling was that they were weak and wicked. The future was ours. If you’d become an MP, like Blair and Brown, in 1983, you wouldn’t have felt you’d be in government any time soon. It was different for my generation. It was much easier. They became spads. They were given safe seats. I remember when Yvette got the safe seat of Pontefract, thinking, ‘Wow, she’s all set now.’ It felt like a great career choice. Now who would think becoming a Labour MP was a great career choice?”

He pauses to order a second coffee. Morgan has a Lancashire accent – he is from Preston – and speaks quickly, and his eyes shine. He is troubled by how unreflective the Golden Generation seemed to him. “They had complete self-belief. They believed that they would be the leadership of the Labour Party. They didn’t hope it. They totally believed it. I wished they’d been more modest, more questioning, more curious.”

In the early 1960s JFK’s young policymakers played squash to keep fit, a departure from their predecessors under Dwight D Eisenhower, who preferred golf. The Golden Generation were similarly obsessed with vigour and competition. Their favoured sport was football, which they played with competence and aggression.

“I remember our team from ITV played Demon Eyes at football,” Morgan said. “We thought they might be this effete bunch. But they were brilliant. They were fitter, faster, stronger. We were out-thought and out-fought. David Miliband, Jamie Purnell –

they were really good footballers. Yes, they were arrogant – but, in a way, they had a right to be.”

***

In his great book The Best and the Brightest, which is about the causes of the Vietnam War and the failures of the young, brilliant and idealistic but flawed policymakers who gathered around Kennedy and Johnson, David Halberstam made a distinction between intelligence and wisdom. That is, between “the abstract quickness and verbal facility which the team exuded, and true wisdom, which is the product of hard-won, often bitter experience. Wisdom for a few of them came after [his italics] Vietnam.”

The basic question that prompted the book, Halberstam wrote, “was why men who were said to be the ablest to serve in government in this century had been the architects of what struck me as likely to be the worst tragedy since the Civil War”.

For some of those who once worked with them, Blair and Brown – so able, so determined, so clearly the best and brightest – are culpable not only for policy disasters such as Iraq and for allowing the Labour Party to be hollowed out, but for the failures of the Golden Generation, whom they nurtured and wanted to succeed them. If political parties are machines for capturing power, it was said to me, they allowed the machine to malfunction.

Blair, unlike Brown, never suffered a general election defeat, and, to the last, he will defend his decision to invade Iraq – the single greatest foreign policy catastrophe since Suez – even if he now concedes mistakes were made in the post-invasion planning. For his part, Brown feels profoundly wronged and misunderstood, and still cannot understand how he lost to Cameron and Osborne, of whom he remains scornful.

“Both Gordon and Tony in their separate ways must shoulder a huge amount of the blame for what has happened to the Labour Party because it was obvious to me ten years ago that we were hollowing out,” a former MP told me. “It was absolutely clear . . . [but] no one was interested. I can’t tell you the degree of apathy, indifference, because every single one of them thought it was going to go on for ever.”

And into these hollow spaces flowed the radical left, activists and members who were disgusted by Blair and by the Iraq War. These activists had never stopped believing their opportunity would one day come again. They continued to organise diligently or cluster in fringe groups or other parties: the Greens, Respect. They wanted change, they believed that moderate social democracy had failed, social media offered them new ways to communicate and connect, and then, presented with it, they seized their chance when events forced Miliband to revise the rules by which the party elects its leader. The party is theirs now. Jeremy Corbyn is their leader.

***

Ed Miliband and I have not spoken since he lost the 2015 election, but it’s clear to me that defeat has changed and humbled David Miliband and Ed Balls. In the process, they seem to have acquired not just greater humility, but wisdom. The Golden Generation were intelligent enough but were they wise enough? Did they really understand the burden and responsibility of elected office, or the struggles and aspirations of the people they wanted to represent? They wanted to lead and they wanted power, but, unlike Blair and Brown, did they know what to do with it? As it turned out, early success was no preparation for winning and holding on to power.

Gordon Brown once told me how his generation had to fight, and fight again, to win the right to control the party. They had to debate and take on the left in smoke-filled meeting rooms, in crowded conference halls, in the public prints, and at rallies. The Golden Generation never had to fight. “They were bright, decent, public-spirited – but they weren’t street fighters,” said Eddie Morgan of his former colleagues. “They were not venal or corrupt but, yes, they lacked fight. But maybe they’re acquiring the fight now – Burnham is going off to Manchester, Yvette and the refugees [she intervened on behalf of Syrian refugees during last year’s leadership contest], Jamie Purnell at the BBC.”

Another former friend who knew the group well said: “Parties in the end are machines for capturing power and there is a sort of life cycle, and you’ve got to be absolutely vigilant about renewing it. Blair and Brown thought they could renew the machine with very clever people, but with one or two exceptions they were – what is the word I’m searching for? – they were servants, they weren’t masters, they didn’t really have a vision of where they wanted to go.”