In his final foreign speech as president, Barack Obama spoke to a crowd in Athens. “As you may have noticed,” he said, “the next American president and I could not be more different. But American democracy is bigger than any one person.” More than a year on, with that proposition tested daily, Obama’s decision to make his last trip abroad to the birthplace of western democracy looks prescient.

The person Obama turned to just before taking the stage was a trim man with thin, close-cropped hair and a furrowed brow who had been at his side on almost every foreign trip he made, and who helped write this and just about every other foreign policy speech the president delivered. Ben Rhodes, the longest-serving member of Obama’s foreign policy team, at the age of 40, has been a permanent fixture in his close orbit; inside the Obama camp, Rhodes was routinely said to be so close to the president that their minds had melded.

A lot of advisers are thinking: how can I get my proposal into this speech? I was thinking: what does Obama want to say?

“It happened relatively quickly,” Rhodes says, attributing it to the fact that he had joined Obama at the start of the presidential campaign, before his own ideas about the world had solidified. “If you are a speechwriter, you have to know what the person you’re writing for thinks. A lot of foreign policy advisers are thinking: how can I get my proposal into this guy’s speech? I was just thinking: what does he want to say?”

As deputy national security adviser for strategic communications, Rhodes was intimately involved in every consequential decision of that era, from the Iran nuclear deal to the restoration of the Cuba-America relationship. But for someone who has played such an important role in the past decade’s geopolitics, his voice and face have been largely unknown. Now, he has a starring role in a new documentary about the last days of Obama’s presidency, shot behind the scenes. The Final Year follows the Obama foreign policy team through the warren of the West Wing (dilapidated, cramped, home to cockroaches and rats), the corridors of the United Nations and on their valedictory foreign trips. Watching it now, knowing the result of the 2016 election and the ferocity of the backlash to come, gives the film a whole new significance.

Photograph: Stephen Voss/The Guardian

“Unfortunately, it’s probably better because of the ending,” Rhodes tells me when we meet in Washington DC, less than half a mile from his former place of work. He is now able to laugh at it, albeit tentatively. “It would have been a nice movie about people working hard on foreign policy, but it acquired this tragic sensibility. It’s a horror movie when you know the twist ending.”

The Final Year focuses on Rhodes and Samantha Power, the former US ambassador to the UN. Both joined Obama’s team from the moment he announced his candidacy, and are the two – sometimes duelling – young guns. Secretary of state John Kerry, the national security adviser Susan Rice and Obama also play major parts, reflecting on the sunset of the administration as the action shifts from the UN headquarters in New York to Austria, West Africa, Vietnam, Laos, Japan and Greenland. Along the way, we see the big set-piece moments – the important speeches, the tense negotiations with other diplomats; we also see the anxieties, disappointments and arguments in back rooms. Above all, the film is a study of the sheer, relentless slog of diplomacy, and its frequently meagre rewards; in September 2016, for example, the team’s hopes of a US-Russian ceasefire in Syria rise with each passing day of relative quiet, and then vanish in a new, entirely senseless wave of violence.

The film is directed by former American television journalist Greg Barker, who has made documentaries about the hunt for Osama bin Laden and the life and death of Sergio Vieira de Mello, a Brazilian UN diplomat killed in Baghdad. The latter was based on a biography written by Samantha Power, a former foreign correspondent, who was Barker’s way into the Obama sanctum. “She [Power] had probably agreed to do this even before checking with anybody,” Rhodes laughs. “We gave Samantha a pretty free run to make her own decisions.” But the president gave Barker’s crew more access than he ever envisioned; the Obama team – nearing the end of the administration – wanted the film to showcase its foreign policy accomplishments, prizes that they fully expected to hand on like polished batons to friends and colleagues in the Hillary Clinton camp.



Ben Rhodes and Obama, 2014. Photograph: Alamy Stock Photo

But as the summer of 2016 turns to autumn, the film shows Donald Trump turning up with increasing frequency, on televisions in airport departure lounges, in conversations with perplexed foreigners. We see blithe certainty turn to niggling doubt, and then, on election night, to devastation; Power and Rhodes, caught on camera at would-be festivities, look as if they have been punched in the gut. (Power hosted a women’s event with Gloria Steinem and the 37 women ambassadors to the UN including Madeleine Albright, who wore a shattered-glass badge in anticipation of Clinton’s achievement.) Rhodes, the wordsmith, has tears in his eyes and struggles to complete a single sentence.

“There was a feeling of, this is not the end of the story that I wanted,” Rhodes recalls. Now, he says, when he looks at the film, “the main thing I think about myself is: you look just tired.” At the time, he was father to a two-year-old; since leaving the White House, he and his wife, Ann Norris, a former senate foreign policy adviser and state department official, have had a second child. He does not miss the adrenaline of negotiating through global crises. “Eight years really was too long. I was only running on fumes the last two or three years. What happens on 21 January [when the Trump presidency began] is that all the adrenaline drains out of you. You cannot get out of bed.”

On board Air Force One, en route from London to France for the 2011 G8 summit. Photograph: Alamy Stock Photo

The 10 weeks of transition from one presidency to the next turned into a painful wake as the seating stands were erected outside the White House for Trump’s inauguration. Today, Rhodes likens it to a scene from a William Faulkner novel. “It’s almost like in As I Lay Dying, when they are building the coffin in front of the mother’s house.”

That feeling dogged Rhodes for the best part of last year, as Obama’s foreign policy record continued to come under ferocious attack. Trump lampooned the Iran nuclear agreement as the “worst deal ever”, hobbling it and constantly threatening to destroy it. Meanwhile, the administration was criticised by both left and right for keeping US forces out of the Syrian civil war, leaving the field to Bashar al-Assad and his Russian and Iranian backers, who flattened entire cities.

During the presidency and since, Rhodes has been a lightning rod for criticism, because of both his combative defence of Obama foreign policy and his closeness to the president. It’s something that clearly still rankles. “I wasn’t even particularly responsible for Syria policy, and I don’t say that to absolve myself. My role was to communicate it. But I was the only one who was willing to go out and defend it. It’s a hard thing to defend when the circumstances are so poor.”

Rhodes is helped with his tie by Obama, personal aide Ferial Govashiri, and director of Oval Office operations Brian Mosteller. Photograph: Alamy Stock Photo

Rhodes joined the Obama team in late 2006, just days after the senator first hinted he might run for president. The 9/11 attacks had persuaded Rhodes to give up his hopes of a literary career five years earlier, and to do something more practical. Jobs as a congressional speechwriter, and spells working for the 9/11 commission and the bipartisan Iraq Study Group, fostered a deep scepticism about the use of US military might as a panacea. In Obama, a resolute opponent of the Iraq war, he found a candidate and a cause.

***

During Obama’s second term, Rhodes spent a significant amount of his time engaged in secret negotiations with Cuban officials in a house laid on by the Canadian government outside Ottawa. There, he and the White House’s Latin America expert, Ricardo Zuniga, launched an ambitious attempt to end a half-century rift with Cuba. “If somebody relatively senior didn’t say, ‘OK, I’ll take this on’, I knew it wouldn’t happen,” says Rhodes, who takes responsibility for the subsequent deal. “I basically said, this is what I want to do in the second term.”

Over a series of eight encounters, the talks expanded from the initial goal of a prisoner exchange to a comprehensive agenda of restoring diplomatic, travel and trade ties. The parties travelled discreetly to Rome for the Vatican’s endorsement as a guarantor, before Obama and Raul Castro revealed the agreement on 17 December 2014.

Rhodes suspects the work of spoilers opposed to the US-Cuban detente. 'I do not believe it is the Cuban government'

In the last days of the Obama administration, Rhodes and the Cuban government worked feverishly to broaden their bilateral ties in order to make the rapprochement harder to dismantle. But a year on, the agreement is under siege. Trump has reintroduced restrictions, and there has been an outbreak of mysterious ailments among US embassy staff in Havana, including hearing loss, dizziness and headaches, which Washington has described as an attack. It has stopped short of accusing Havana of carrying out the attack, but holds the Cuban government responsible for failing to protect US diplomats. All but a skeleton US embassy staff have been withdrawn from Cuba and most of the Cuban diplomats in Washington have been expelled. The cause of the health issues afflicting the US diplomats remains unknown.

Rhodes, however, suspects the work of spoilers opposed to the US-Cuban detente. “I really do not believe it is the Cuban government,” he says. “Whether it’s a third party like Russia, or whether that is some harder-line faction in Cuba, to me it is someone with a motivation to kill the relationship or set it back. And unfortunately, they’ve succeeded.”

At the heart of The Final Year is the dilemma that has haunted every US administration: under what circumstances should it use its overwhelming military might abroad? Both Libya and Syria were “wars of choice”: that is, where there was no direct threat to the US population or homeland, but where long-term global security, the use of chemical weapons and the mass slaughter of civilians were at stake.

Obama and Samantha Power, US ambassador to the UN. Photograph: Pete Souza/The White House

In Libya, the administration opted to go to war, albeit in an ancillary role. But Obama decided not to send US forces into battle in Syria. That decision nearly tore the Obama team apart, putting Power, a passionate advocate of humanitarian intervention, at odds with Rhodes, who channels the president’s innate caution. On screen, the strain is clear. “The two powerful threads in American history – the costs of action and the costs of inaction – come together,” Rhodes says, defending the decision to allow this tension to play out in front of the camera. “I thought it was relevant to show there were different points of view, and I say that with no certainty that one point of view is right. Samantha – I won’t speak for her, but her whole career has been about the ghosts of those who suffered when there was not intervention.” Power declined to comment for this piece.

Rhodes believes Obama was held back by other phantoms, of those killed in previous US military adventures abroad. One of the trips the president chose to take in his valedictory year was to Laos, which the Nixon administration devastated in its cross-border pursuit of its Vietnamese enemies. “Laos is the ghost of American military interventions past,” Rhodes says. “We definitely felt those ghosts as we made decisions.”

Obama came under great criticism over Syria; for declaring that the Assad regime’s use of chemical weapons would be a “red line” for US military action, and then failing that test by not striking after a mass-casualty chemical attack in August 2013. Rhodes argues that the conditions for effective action – congressional and allied support – were not there. The British parliament’s vote against taking part in airstrikes was “huge”, he says, when it came to taking a final decision.

The question hanging over the current administration is whether they have rational discussions about any of these issues at all. As I sit down with Rhodes, Washington DC is consumed by the revelations in Michael Wolff’s new book Fire And Fury, depicting chaotic scenes inside the Trump White House, with rival factions consumed with infighting, and even close associates calling the president an “idiot” or childlike.

Nothing suggests the Trump White House has a process to deal with something like Ebola

I ask Rhodes what he made of it, having been in those rooms himself. “There are a very limited number of people in senior roles at the White House, and time is their most precious asset,” he says. “Distractions like this book consume people’s time, focus and emotional energy. Every minute spent responding to, meeting about or thinking about a controversy like this is time not spent on something else.” He adds that the Trump administration has yet to confront a major international crisis not of its own making. “You cannot respond to a crisis without a good process, and I have read and seen nothing that suggests the Trump White House – even under General [John] Kelly [Trump’s second chief of staff] – has a process that is suited to deal with something like, for instance, Ebola.”

But it is possible that, while the Obama administration upholds Athenian ideals of democracy, the outlook of much of the electorate has more in common with Sparta. “It’s difficult to make diplomacy sound as attractive as taking out a dictator with our military,” Rhodes says.

He points out that the wisdom of the Iran deal, in which Tehran accepted strict curbs on its nuclear programme in return for sanctions relief, was unquestioned in other signatory countries: the UK, France, Germany, Russia and China.

“Talking and diplomacy is often seen as a concession in America, in a way that it is not in other places,” Rhodes says. It has also been argued that the Obama administration was much better at winning hearts and minds abroad than it was in the American heartland, where the president’s professorial mien came across, or was easily caricatured, as aloof.

In an interview with the New York Times in 2016, Rhodes said of the American press that they “literally know nothing” and described the Washington foreign policy establishment as “the Blob”. Today, he says he regrets making dismissive remarks about the press, but stands by his use of “the Blob” to describe the thinktanks along Massachusetts Avenue in Washington, which he believes still err on the side of military force. Trump’s scaling up of the military presence in Afghanistan and bombing of a Syrian airfield have been welcomed there, he argues.

In the Oval Office, working on Obama’s speech to the American Israel Public Affairs Committee, 2012. Photograph: Alamy Stock Photo

Today, Rhodes continues to work for Obama, travelling with him on all his many foreign trips in the fading afterglow of the presidency. A secret service detail still flies with them, Obama is still addressed as president, and heads of state make a point of meeting him – often with pointedly more enthusiasm than they offer to his successor. For an embattled liberal world, Obama remains the king in exile, trailed by crowds nostalgic for happier, more hopeful times.

“What’s weird about Obama is that he’s so popular,” Rhodes says. “There will still be hundreds of people in front of the hotel. The demand for him to be visible and to offer hope and consolation is extreme, and his capacity to do so in the role of former president is inherently limited.”

But he insists that the former president remains unruffled by the wrecking ball Trump has taken to his legacy, from walking out of the Paris climate accord and a trans-Pacific trade partnership, while threatening (though not quite killing, as yet) a 2015 nuclear deal with Iran. Progress does not follow a straight line, Obama has frequently observed, always adding that it trends upwards. “He is a pretty disciplined person,” Rhodes says. “He is this guy who takes a sweeping view of history. He has a serenity that I don’t have. I get more exercised.”

Like his boss, Rhodes is hopeful the political pendulum will swing back after straying into the territory of the hard right, but he admits to doubts. “I do worry that the basic standards of how the presidency operates are being eroded. The pace at which the state department is being gutted – I really worry about that. I don’t know how you train multiple generations of diplomats.” If Trump stumbles into a war in the Pacific by miscalculating his personal brinksmanship with Kim Jong-un, talk of pendulum swings in US politics might be beside the point. As Rhodes says, “We would be in a different reality.”

• The Final Year is in cinemas and on iTunes from 19 January.

• Commenting on this piece? If you would like your comment to be considered for inclusion on Weekend magazine’s letters page in print, please email weekend@theguardian.com, including your name and address (not for publication).