For many people who have followed the career of Boris Johnson – EU diplomats past and present, as well as European officials and journalists – his return to Brussels on Thursday as prime minister, to proclaim a deal that would end the UK’s EU membership, was not just an event of obvious historic importance.

It also had an unreal quality – bringing a swirl of memories flooding back of how the “Boris story” had all begun in Brussels three decades before. As what was likely to be the last EU summit with the UK in the fold got under way in the early afternoon, many remarked how difficult it all was to process.

Johnson, the ex-journalist and now the statesman, appeared next to European commission president Jean-Claude Juncker, whose views he had mocked in newspaper columns and speeches since the late 1980s, the two now all smiles and best of friends. “Jean-Claude’s the boss here,” Johnson joked as they trumpeted their deal.

The former foes congratulated one another on an arrangement that Johnson said would at last set the UK free to “take decisions about our laws, our borders and our money” as an independent nation state. Juncker – the arch-enemy federalist in the eyes of Eurosceptics since the Maastricht treaty days of the early 1990s – followed up by saying he was happy at their joint success, but sad the UK was leaving. But he was also clear that this really was it.

Doing Johnson a favour, he upped the ante on Westminster MPs by insisting there would be no further extension for the UK if the deal was voted down. “This deal means there is no need for any prolongation,” Juncker declared. It was the Johnson deal or no deal.

Like other EU politicians of longstanding, Juncker first came across Johnson when he was a young Brussels correspondent, making his name on the Daily Telegraph. Then he was the journalistic bane of the commission and every true believer in the European project.

As he watched the prime minister on Thursday, Geoff Meade, the former Press Association correspondent in Brussels who once worked alongside Johnson, recalled the numerous “fanciful” stories from Brussels that had appeared on the Telegraph’s front page between 1989 and 1994, including one in 1991 about the commission having to be blown up because of dangerous asbestos. It was a Johnson exclusive, which he followed up with another about plans to replace the building with a giant skyscraper so high that planes would have to be diverted to avoid crashing into it.

“There were more than a few people at the summit on Friday observing the scene and pondering the Johnson journey from sometimes hapless hack, peddling half-truths and fanciful interpretations of the deeds and motives of “unelected Eurocrats” to powerful PM, now leading a proud nation out of the European club on the back of EU paranoia he helped foster 30 years ago,” said Meade.

As he addressed journalists at just before 7pm on Thursday, having received the formal approval for the deal from the other 27 EU leaders, Johnson looked exhausted. Showing deference to the moment, he struck a sober, serious tone and chose an oddly constructed metaphor to describe “exciting” prospects ahead for the UK once it had left the EU. “The extraction having been done the building can now begin,” he said.

It had been a trying few days. The chief stumbling block all along had been the problem of the Irish border: how to satisfy Johnson’s demand that the UK must leave the EU customs union while at the same time meeting the EU’s and Irish red-line demands that there be nothing that could be construed as a hard border on the island of Ireland itself.

Facebook Twitter Pinterest Johnson in talks with Ireland’s taoiseach Leo Varadkar in the grounds of Thornton Manor hotel near Birkenhead last week. Photograph: Getty Images

For Dublin, London and Brussels, the nadir was reached last Tuesday, with the now notorious phone call between Johnson and Angela Merkel. The German chancellor rejected Johnson’s plea to revive proposals that had already been rejected by Brussels. By the end of the call, Merkel is said to have become blunt, concluding that the Irish border left the UK with a major issue it could not wish away.

Just as disastrous was the aggressive briefing of the call by Downing Street, whose officials said that it was a “very useful clarifying moment” because it suggested a deal was “essentially impossible, not just now but ever”. That frustrated many in Brussels, who assumed the deal talks were over and the blame game had begun.

In Ireland, there was also pessimism and dismay at “the tone that was coming out” of No 10 by Tuesday night. It seemed impossible that a deal could be done in just a few days. Even as officials headed to a meeting between Johnson and Leo Varadkar at the neutral venue of Thornton Manor in the Wirral, hopes were not high.

“Even as we travelled to Liverpool, I’m not sure any of us were feeling particularly hopeful and there was certainly no expectation that we would be going home that evening with something as positive as a pathway to a deal,” said an official.

Yet hopes began to rise when the meeting between the leaders, scheduled for 30 minutes, dragged on and on. So prolonged was the discussion and subsequent walk in the garden that their hungry teams began to worry about whether it would be OK to eat their sandwiches before their respective leaders had eaten anything.

So what changed? “I guess there was a reality to the discussions,” said a diplomatic source. “This could work and this can’t.” In truth, the Wirral meeting had been the centrepiece of a wider diplomatic effort. Simon Coveney, the Irish foreign minister, had been in Brussels the evening before, clarifying what could and could not be accepted. There was a key conversation between the chancellor Sajid Javid and his Irish counterpart Paschal Donohoe on customs issues, which may have given a sense that the UK was prepared to make a move.

Meanwhile, Varadkar’s hard line that the only workable solution was for Northern Ireland to stay in a customs union with the EU had begun to change, almost imperceptibly. By the time he and Johnson met, some diplomatic sources said they believed the Irish leader’s mind was shifting, and that he also had had word from the UK side that Johnson was ready to move too, and was looking for a possible landing zone for a deal.

Wednesday opened with hopes raised but many contradictory messages. The EU’s chief negotiator, Michel Barnier, had set an 11am deadline for a legal text on any new deal to be submitted but this deadline passed yet again with issues over the Irish border still unresolved.

MPs were shuffling and out of Downing Street; Tory Eurosceptics, Labour, and DUP alike. All the while the clock was ticking. Johnson had to have a deal that could be agreed by parliament by 19 October or he would have to ask for a Brexit extension under the terms of the Benn act.

On Wednesday morning, Brexit secretary Stephen Barclay appeared before a Commons select committee and said that the PM would write a letter in accordance with the legislation if no deal had been agreed by the deadline. But Downing Street was still suggesting that while Johnson would obey the law, the UK would leave on 31 October come what may. A senior Labour frontbencher said: “It’s just games, more Johnson games.”

Both could not be true, but government officials would not say more. Johnson met his cabinet at 2.30pm on Wednesday but full details of the deal he was negotiating were not given because it had not been finalised and the legal text remained a work in progress between London and Brussels.

At 4.30pm Johnson attended a meeting of the 1922 Committee of Tory backbenchers, but only stayed for 10 minutes saying it was “time to get Brexit done”, and repeating Theresa May’s mantra that it would be the deal on offer, or no deal. Afterwards, while there was growing optimism no one knew if a deal (whatever it would be) would gain the necessary support from the DUP, pro-deal Labour MPs, Tory hard Brexiters in the European Research Group, and former Tory Remainers who had lost the party whip to get it through parliament.

He would have loved being one of the principal actors, instead of one of the most irreverent reporters Andrew Gimson, biographer

Overnight, however, the DUP pre-empted Johnson’s big moment – the announcement of a deal about which its leaders had been in talks with Downing Street for days. The landing zone Johnson had agreed with Varadkar would, to the DUP’s dismay, involve a customs border in the Irish sea and an end to the DUP’s veto over the future arrangements in the Stormont assembly. Nigel Dodds, the DUP’s leader at Westminster, said Johnson had put “far too much” effort into getting a deal and far too little into ensuring it did not sell Northern Ireland down the river. The DUP would vote against the deal.

At 9am before he set off for Brussels, Johnson made a decision to press ahead despite the DUP’s objections. He was going for broke, ready to gamble on winning over enough Labour Leavers, independents and former Tory MPs to get it across the line in parliament. He called Juncker to finalise the deal before announcing it on Twitter.

When he appeared with Juncker soon after, the commission president’s insistence that there would be no extension came as a bolt from the blue. If that view was shared by the 27 EU leaders, it really would be the Johnson deal or no deal. Momentarily it seemed the tactical game was being won by Johnson and Downing Street.

Quickly, however, it became clear that Juncker was merely lending his new ally Johnson short-term tactical support. In the leaders’ Brexit discussions that followed, Merkel told Johnson that if the deal was rejected by the Commons, the EU would have to seriously consider an extension. It was the opposite of what Juncker had said. Delay was still possible. “What matters is not what Juncker says but what the 27 leaders say,” said an EU diplomat. Just as fast as some clarity and clear advantage to one side seemed to have emerged, the fog of doubt about where it was all going had descended again.

At 5.30pm, Varadkar, the EU council president Donald Tusk, Barnier and Juncker appeared and declared the deal sealed. Tusk seemed weary and emotional, saying he hoped the UK would return one day. Varadkar likened the moment to bidding farewell to a dear old friend who was going on a long journey into the unknown, and hoping it would all turn out well. There was no repetition of Juncker’s comments about no further extension.

Minutes later Johnson appeared for his press conference, declaring he was “very confident” he would have the numbers in parliament to get the deal through. He also suggested that more reassurance would be given to Labour and other MPs before Saturday’s vote on issues such as workers rights and the environment. But he had not anticipated the successful move led by former Tory minister Oliver Letwin to force a further delay.

Hopes of a deal that could pass through parliament have risen and fallen yet again – but as ever with Brexit, the final outcome is still unclear. Andrew Gimson, author of the biography Boris: The Making of the Prime Minister, said after observing the PM’s appearance alongside EU leaders that despite the dramas and continuing uncertainties, Johnson would “have loved being one of the principal actors, instead of, as he was three decades ago, one of the best known and most irreverent reporters” even though he had yet to achieve the Brexit he had done so much – in his many different roles – to advance down the years.