As a full moon rose over the London Stadium, athletics’ greatest pantomime villain, Justin Gatlin, sank his teeth into the carefully laid plans for Usain Bolt’s retirement party. And then, in the last desperate steps of a compelling world championships 100m final, brutally ripped them apart. The 35-year-old American, who was banned twice early in his career for doping offences, is not necessarily the champion the sport wants. But given its problems, it is one that many will feel it deserves.

When it flashed on the scoreboard that Gatlin had taken gold in 9.92sec – 0.02 clear of the young American Christian Coleman, who took silver, and Bolt who claimed bronze a further 0.01sec behind – most of the 56,000 crowd in the stadium went silent before they collectively booed in disgust. A couple of years ago, Gatlin described himself as “the Batman of the track – a vigilante”. But few in the London Stadium were celebrating the rising again of this self-styled Dark Knight.

World Athletics Championships 2017: Justin Gatlin beats Usain Bolt to 100m gold – as it happened Read more

It was, staggeringly, the US sprinter’s first world title since 2005. It has been a topsy-turvy journey since then – which included a four-year doping ban in 2006, which he claimed came from a massage therapist rubbing testosterone cream on his legs without his knowledge. Everyone thought Gatlin, a 25-1 underdog going into the final, was too old and too slow.

Instead Bolt’s main challenge was supposed to come from the 21-year-old Coleman, who gave up his dreams of playing American football to move to track and field. Yet somehow Gatlin found a way to roll back the years – however much it infuriated the crowd.

Bolt’s reaction to his defeat, his first in a major final since 2011 when he false-started in the 100m in Daegu, showed the immense class of the man. Immediately he went to hug Gatlin, who bowed to him, to congratulate him on his performance. Then he also lavished praise on the crowd. “London, I really appreciate the support you gave me. I’m just sorry I couldn’t deliver as I wanted. It is one of those things.”

And despite suffering the most painful defeat of the year, he turned the atmosphere from a morgue to a party by performing his traditional lap of honour and posing for selfies as if he had just claimed his 12th world title.

Afterwards Bolt was left ruing his shocking start, which left him two metres behind Coleman and a stride behind Gatlin after only 30 metres – and ultimately with too much to do. The way he rises from the blocks has never been much better than adequate, even at his peak. But in London it has resembled a hospital patient getting out of bed following back surgery. “My start is killing me,” he admitted. “Normally I get better through the rounds but it didn’t. This is the first time in a major championships it hasn’t come together. And that’s what killed me. It was the reason I lost.”

Two years ago, at the world championships in Beijing, Bolt and Gatlin had gone head to head in an epic showdown. With 20m to go the two men were locked together in a desperate tango, stretching and straining for the line. But then Gatlin over-reached, stumbled and, in a flash of 50,000 camera-phones and a whoosh of cheers, Bolt had him – just.

This time Gatlin kept his composure. After 70 metres it looked like Coleman, who made a superb start in lane five, was going to claim his first gold medal. But with the hot breath of Bolt on his shoulder, he began to tie up, allowing his compatriot in lane eight to pounce.

As each of the eight finalists were introduced to the sound of a thumping heartbeat, Bolt had smiled and put his fingers to his lips. Coleman puffed out his cheeks, trying to blow out the tension. Gatlin, meanwhile was stoic and focused in the face of sustained booing. Then, as the sprinters sank into their blocks, the chants began to grow: “Usain Bolt! Usain Bolt!” It was reminiscent of the crowd in Zaire chanting “Ali Bombaye!” before his bout with George Foreman in 1974. However, to the shock of just about everyone, once again the underdog applied a knockout blow.

No wonder the crowd were stunned. After all, Bolt’s record since he blitzed and charmed his way into the public’s consciousness at the Beijing Olympics, when he obliterated the 100m and 200m world records, has been staggering.

Facebook Twitter Pinterest Usain Bolt embraces Justin Gatlin after the final of the men’s 100m final. Photograph: Tom Jenkins for the Observer

Before Saturday night’s semi-finals and final Bolt had run 142 races since the start of 2008 – and won all but seven of them.

Yet throughout 2017 there had been the nagging feeling that he was uniquely vulnerable. He had missed three weeks of the season after his close friend Germaine Mason was killed in a road accident. His back had troubled him. His 100m times were no longer immortal. The door was open for someone to take him. The question was, could anyone step up?

That was answered in the semi-finals as Coleman surged out of his blocks and held his form and his nerve to win in 9.97sec, with Bolt, who again made a wretched start, 0.01sec back.

It was Bolt’s first defeat for more than four years, since being beaten by Gatlin in Rome in June 2013. With no other athlete breaking 10 seconds, the final looked set to be an immediate rematch between the sport’s greatest sprinter and showman and the young contender. Gatlin, however, had other ideas.