Barack Obama’s second term began in earnest not with his re-election, nor subsequent inauguration, but on a dark December day exactly halfway between the two.

The massacre of 20 children at Sandy Hook elementary school in late 2012 was – according to one aide with him in the Oval Office when Obama was told – to prove the worst moment of either term. “You think this job is so powerful, but it’s not,” a stunned president is said to have observed as he received news of the shooting in Newtown, Connecticut.

Not even revulsion on this scale could provide enough political capital to prevent mass shootings from punctuating Obama’s presidency. Failure to push gun control legislation through the Senate in those first weeks after re-election was to prove a foretaste of many such frustrations over the following four years.

Some wounds were self-inflicted. Administration loyalists concede the most difficult period of the presidency was the bungled rollout in October 2013 of online healthcare exchanges promised under the Affordable Care Act (ACA). The fact it was nicknamed Obamacare by everyone only heightened the embarrassment.

Even the widely celebrated achievements of the second term were more diffuse and distant than before. Where the first four years involved a domestic economy rescued from the depths of recession and Congress tamed amid passage of the ACA, the successes that followed were often overseas and incomplete: diplomatic rapprochement with Cuba, a climate change agreement in Paris and a deal to contain Iran’s nuclear potential.

Reporting for the Guardian on this difficult second act was complicated by our own walk-on part in the drama. A conference call in June 2013 with White House national security staff about the bulk collection of mobile phone records was the first of several awkward conversations involving a young whistleblower called Edward Snowden.

Barack Obama: I'm 'confident' I could have won a third term as president Read more

“I’m not sure being handed a bunch of classified documents qualifies as ‘breaking’ a story, but I certainly acknowledge that Snowden has used the Guardian as his principal outlet,” wrote press secretary Jay Carney in December 2013 when we asked to pose questions more publicly at a year-end press conference dominated by the affair.

Despite the frosty reception, the Guardian secured its first permanent seat in the press room in the second term, and a spot on the press corps’ pooled reporting rota, which assured a bird’s eye view whether we behaved ourselves or not. Mostly this was in or near Washington, for routine events such as Obama’s hosting Prince Charles in the Oval Office or the Gulf monarchs at Camp David.

But we were there too, in the pouring rain, when the Obama family toured the streets of old Havana during the first US presidential visit to Cuba since 1928, and there again in the rain when he became the first president to attend India’s Soviet-themed Republic Day parade. Shouted questions, delivered from Mexico or the South Lawn, even received occasional answers.

Not every pool duty went to plan. We were also the first newspaper on Donald Trump’s campaign plane for a short-lived experiment in letting reporters travel with him and the last to shadow Hillary Clinton on the day she lost the election. On board Air Force One, before Obama’s equally disastrous midterm elections in 2014, things got so bad the president came back to deliver cake for reporters to avoid exposing his press spokesman to another depressing briefing on Halloween.

White House reporting also provided unexpected glimpses of domestic life: waiting in press vans while Obama attended his youngest daughter’s dance recital or finished the weekly ritual of 18 holes of golf at Andrews air force base. Lest anyone assume such perks made up for life lived in a goldfish bowl, the Obamas would host a Christmas party for journalists at which they spent the whole night “gripping and grinning” for posed photographs. Travelling in the motorcade was a reminder that presidential life was a permanent blur of motion-sickness and instructions not to open the windows.

Few reporters could ever boast they felt truly at home in the Obama White House. After a meeting in the West Wing with Carney’s successor, Josh Earnest, I took a wrong turn down the corridor which I thought led back to the press room only to be flattened against a wall by a secret service officer when I nearly bumped into Potus emerging from the Oval Office.

The hardest stories to report on in the second term were those involving the Middle East, where Obama’s foreign policy could veer unpredictably almost by the hour. What looked to be a determination to launch imminent airstrikes against the Syrian government in August 2013 turned out to be nothing of the sort. The president had taken an evening “walk-and-talk” around the garden with his chief of staff and surprised everyone by deciding instead to seek congressional approval – knowing, as in so many other instances, it would not be forthcoming.

Obama himself found the relentless Republican opposition strangely liberating. “It’s a theme that’s continued throughout my presidency,” he said recently. “The degree to which Republican critics could be on every side of every issue, depending on what decisions I had made. So if I was initiating military action, then the criticism was: ‘This was irresponsible’ or ‘Why’d you do it this way?’ If I didn’t, ‘You’re weak.’”

At times the deadlock looked worse than it really was. During one of several standoffs over the federal budget, journalists camped out after midnight on Capitol Hill waiting for the government to shut down, but the House speaker, John Boehner, was secretly back at 1600 Pennsylvania Avenue holding private talks. White House officials were instructed to use the time-worn deflection “we have no scheduling announcements to make” when asked if they would be meeting, because the speaker’s office had demanded secrecy.

The partisan gridlock wrought undoubted damage, too: both to the reputation of mainstream Republicans and the legacy of a restless president who was elected on the promise of becoming a change-maker who would heal such wounds.

It has left Obama particularly exposed to the arrival of an even more intransigent Republican in the White House this month. Forced to rely on executive actions to complete the modicum of domestic policy reform that was achieved or anchor progress on climate change, the 44th president of the United States is uniquely vulnerable to a 45th president who has pledged to reverse much of it with an equally swift swish of his pen.

Obama's legacy: Lorrie Moore, Richard Ford, Marilynne Robinson and others look back Read more

Donald Trump may nonetheless unwittingly burnish his predecessor’s reputation – simply by providing the most dramatic character contrast imaginable. While the next president has shown himself to be easily riled and goaded, Obama’s steadfast response to crises now seems more statesmanlike than indecisive. Many Americans already miss the ineffable cool of a law professor. Never one to speak in soundbites, the president’s ponderous circumlocutions look positively Churchillian compared with his successor (or predecessor).

Like George W Bush, Obama will also be remembered for the all-too-human expression on his face when confronted with the horrors of the job. It took some time for a full set of photos to emerge of the moments the previous president learned of the September 11 attacks. Obama’s time in office was more frequently documented by the in-house photographer Pete Souza, whose carefully controlled access irritated independent journalists but provided many of the most memorable images of any presidency. Souza’s snap of Obama listening to his counter-terrorism adviser John Brennan tell him about Newtown immortalises the instant the most powerful man on earth realised how little power he really had.