Three years ago Eddie Jones surprised me for the first of many times. We met to discuss how we would work together on a book which was still a shadowy idea in the autumn of 2016. In those strange early days of our collaboration I was still trying to figure out Jones. I had heard stories about his abrasive coaching style, which had apparently left a trail of players weeping and hiding under tables from Canberra to Tokyo. I had also heard his whiplash quips and spiky insights because, as England coach, Jones’s press conference highlights were never boring.

Jones arrived in England rugby with a bang on 1 December 2015. England were in chaos after being knocked out in the group stages of a World Cup they had just hosted. Jones, who had masterminded the game of the tournament by inspiring Japan to victory over South Africa, replaced Stuart Lancaster as head coach. He achieved his first goals in startling style. England won the 2016 Six Nations grand slam and all three Tests against the Wallabies in his native Australia – and every match of the 13 they played under Jones in his first year in charge.

Amid the praise, Jones overturned the first of my many preconceptions. After he had explained how he prepared his teams, using the example of Japan’s shock defeat of the Springboks, Jones said something totally unexpected: “Mate, until that whistle blows, you don’t know what’s going to happen.”

Jones had told me how he had changed the culture of Japanese rugby with meticulous detail, and run a training camp to “Beat the Boks” which was as precise as it was ferocious. Yet, suddenly, he revealed that his intricate planning was ultimately shrouded in uncertainty. I was surprised but intrigued.

The more we spoke, the more I understood what Jones meant. For all its brutality, rugby can be a delicate game of agonisingly fine margins. A player’s confidence can be fleeting and, if it slips once the whistle blows, there is little a coach can do. After the game starts Jones’s ability to influence the outcome is minimal. He relies on his players to make good decisions, to think and adapt. As a coach all he can really do in the midst of a game is to be smart in his use of substitutions.

“That’s why preparation is everything,” he said. “If you get it right, the chances of your team being successful are high. But in a team of human beings nothing is guaranteed.”

In the mud and muck at Randwick under Bob Dwyer, who later told a heartbroken Jones he would not be good enough to play for the Wallabies. Photograph: Peter Rae/Fairfax Media via Getty Images

Jones spoke vividly of how, shortly before Japan faced the mighty Springboks, he had no idea what would happen. After months of hammering home the message to his assistant coaches and players that they would beat South Africa, he was besieged with uncertainty. He was very quiet as the team bus rumbled slowly to the ground.

Japan had won only one World Cup game in their entire history, 24 years before against Zimbabwe. The Springboks had been world champions in 1995 and 2007. Japan had lost every other World Cup match by an average of 48 points – with the most humiliating defeat being in 1995, when they conceded 145 points to New Zealand.

His vulnerability and doubt were fascinating. It made Jones very human and much more interesting than the cartoonish persona of a hardboiled tyrant that some people foisted on him. He then leaned forward and explained the terrible beauty and consuming pressure of elite sport: “This is why we do it, mate. This is why we don’t put on a suit every morning, pick up a briefcase and catch the same eight o’clock train into a routine office job. This is the feeling you get nowhere else in life. It’s this intensity, this fear, this hope, this thrill all knotted up in your gut. There is a need for courage in the face of adversity. This is what we do.”

Jones was a teacher before he became a coach. The two vocations, for him, mirror each other. He loved teaching young players to steel themselves and to keep their heads clear and their hearts steady amid the heat of battle.

England were on a roll which would eventually end in a world-record equalling run of 18 consecutive Test victories. Jones waved a cautionary finger and surprised me again. He warned that there would be dips and setbacks. “Wait until the third season, mate,” he said. “We might hit a bit of a low then.”

Jones had reviewed the records of southern-hemisphere coaches who had worked in Six Nations rugby. They all suffered a third-year dip. After a honeymoon first season, in which they radically improved the fitness of their northern-hemisphere team, they continued to drive up skill levels in year two. By the time the third season unfolded the southern-hemisphere lift paled as familiarity set in and standards stagnated. This was a four-year project, Jones stressed, and the way he dealt with a third-year slide would shape England’s World Cup chances in 2019.

With his Australia players after losing the Rugby World Cup final to England in 2003. Photograph: Peter Parks/AFP/Getty Images

I raised a brow – but in 2018 England hit the buffers under Jones. They lost five matches in a row. His role as a messiah was replaced with depictions of Jones as a clown or an Aussie bastard. I saw him relatively often during this period and he was always calm.

We revisited his 2016 prediction of the third-season blues but he had no interest in making any “I told you so” speeches. He was embroiled in the intricacies of coaching. His best players were fatigued after the Lions tour of New Zealand in the summer of 2017. But he had chosen to drive them harder than ever in training that autumn and again before the 2018 Six Nations. Jones wanted to put them under greater stress than ever before. He knew it might cost England in the short term – but they would be stronger and harder when it mattered most in 2019.

I saw him soon after England had finished second-last in the 2018 Six Nations. Ireland had hammered his side at Twickenham to win the grand slam and the rugby world was certain. England, under Jones, were shot; Ireland were the new joint favourites alongside New Zealand for the World Cup. “We’ll be all right, mate,” Jones said. “The plan is on track.”

I was beginning to understand how Jones worked and planned ahead. He had shared other memories which proved that, despite the uncertainty he felt just before every big game, results usually went the way he predicted. There was one striking example. Jones and Australia were ridiculed after losing 50-21 to New Zealand in Sydney in July 2003. It was a grim outcome; but Jones felt encouraged rather than crushed. He had seen how Australia could beat the All Blacks in a World Cup semi-final three months later.

He drilled his team in a new plan which would dilute New Zealand’s counterattacking game. Jones felt certain of victory and, as he expected, Australia cruised into the World Cup final with a 22-10 win. The rugby world was shocked. But his players had stuck flawlessly to the strategy and it was the most comfortable the Wallabies had ever looked against the All Blacks.

Beyond the rugby I was immersed in the story of his life. Jones is resilient and tough, and he believes these qualities were passed on to him by his mother. The grit of Nellie Jones, who is still alive and well in Sydney at the age of 94, was forged in the second world war. After Pearl Harbour was bombed by Japanese planes in December 1941, she and her family were sent to separate internment camps – despite the fact that, as a 16-year-old girl in a Japanese family, she had been born and raised in California. President Franklin D Roosevelt was candid in referring to the relocation centres as concentration camps.

Getting his hands on the Webb Ellis Cup with the Springboks in 2007 after being hired as a consultant. Photograph: Jason O’Brien/Action Images

Nellie did not see her dad for four years. After the war she wanted to return home, to the orange orchard her father owned outside Sacramento, but her dad was embittered. He moved the family to Japan and they lived near Hiroshima, which had been devastated by the US atomic bomb on 6 August 1945. It did not help that Nellie looked Japanese and spoke the language. The locals knew she had been born in America and, at the age of 21 in 1946, Nellie felt like an outsider in Japan.

Eventually she began working as an interpreter for the British Commonwealth Occupation Forces. Their headquarters were in Eta Jima, 20km from Hiroshima, and it was here that she met Ted Jones, an Australian soldier. They soon fell in love. In 1947 Corporal HJ Cooke had become the first Australian soldier to apply for permission to marry a Japanese woman and bring her home. The application was rejected after Arthur Caldwell, Australia’s minister of immigration, refused her entry, saying: “It would be the grossest act of public indecency to permit a Japanese of either sex to pollute Australia.”

It took until March 1952 for Japanese women to be allowed to enter Australia with their husbands. By 1956 Ted and Nellie Jones were one of 650 married couples that left Japan for Australia. They settled in Tasmania and Nellie faced yet more prejudice. Yet Nellie and Ted brought up their children, Diane, Vicky and Eddie, in a way that spared them this pain. They were raised with a sense of freedom and happiness.

Jones began to understand his mother’s past only when she gave him a copy of David Guterson’s Snow Falling on Cedars – a novel about a young Japanese-American man accused wrongly of killing a white fisherman. His trial plays out against the anti-Japanese sentiment that coursed through America in the wake of Pearl Harbour and the war. Nellie did not talk about racism or suffering. Instead she used the novel to illuminate her past.

It was moving to hear this story and it helped Jones to talk more personally. He described his dad, Ted, as an uncomplicated Aussie, the best kind of larrikin who worked hard and enjoyed watching sport and having a couple of beers with his mates on the weekend. But something deeper stirred in Jones when we discussed his dad. His eyes often welled up with tears and he could not speak. Despite his loss, I think Jones was touched profoundly because he remembered how lucky he had been growing up in Sydney as Ted and Nellie’s boy.

Eddie Jones as a young boy.

He never felt Japanese but from a young age Jones sensed that sport was his path to acceptance in Australian society. Jones could only remember one incident of racism in his playing career. He was playing for New South Wales against Queensland and their hooker called him a “Chink” and a “Chinese bastard” “Mate,” Jones replied, “you’re too stupid to know the difference between the Chinese and the Japanese.”

His life had been shaped by a fortunate coincidence. On his first day at kindergarten in the working-class Sydney suburb of La Perouse, Eddie sat next to Mark, Glen and Gary Ella, three Indigenous Australian brothers. Mark and Glen were twins; Gary was a year younger. Eddie and the Ellas would stick together through primary and high school. The Ella boys were just three of 12 children who lived with their parents in La Perouse – which was called “the Soweto of Sydney”. There was no inside toilet nor any hot water in the Ella’s two-bedroomed house but it rocked with laughter, love and life.

Eddie and the Ellas went to Matraville High and played for, arguably, the greatest Australian schoolboy rugby team in history. The school was located in the shadow of Long Bay jail – the most notorious high-security prison in Australia. Both the school and the jail carried a disproportionately high percentage of young Indigenous people.

Jones suggested I interview Bob Dwyer. Apart from steering Australia to victory over England in the 1991 World Cup final at Twickenham, Dwyer had been Jones’s mentor after coaching him for years at Randwick Rugby Club. Dwyer had also been in the crowd on the famous day when Matraville beat St Joseph’s, the most prestigious rugby school in Australia.

Dwyer told me it looked like a Hollywood movie as the contrast between St Joey’s and Matraville was comical. The privately educated boys resembled a well-drilled team of all-American jocks. Matraville’s small backs were called “the blackline with a red tip” because they consisted of six Indigenous kids and a red-haired winger, Greg Stores. Glen and Mark, Eddie’s best friends, sported big black afros which would have fitted more funkily in the Jackson 5 than an Australian schoolboy rugby team. Eddie Jones was their tiny half-Japanese hooker.

Joey’s came out fast, like superstars, playing slick and aggressive rugby. But Matraville tackled hard and low, and ran at them relentlessly. Their backline was ridiculously flat as they passed the ball in dizzying patterns. Their joy and skill, particularly the Ellas and Lloyd Walker, a silky smooth Indigenous centre, astonished Dwyer. The four Indigenous boys and Jones joined Dwyer at Randwick – and all five of them, a third of the Matraville team, competed in international rugby years later. Jones, of course, did so as a coach.

Dwyer broke the news to Jones that he would not play for Australia. He was Australia’s coach in 1990 when it seemed certain he would select Jones as the new Wallaby hooker. Dwyer told me Jones looked sadder than anyone he had ever seen when he heard the news. Dwyer had chosen Phil Kearns, his young understudy at Randwick. Jones was heartbroken.

His anguish was accentuated by the fact that he knew he was not really good enough to play Test rugby. He was far too small. Kearns was six foot tall and weighed 17.3 stone. So he was four inches taller and four and a half stone heavier than Jones. Kearns went on to play 67 Tests for Australia and was part of two World Cup-winning squads, in 1991 and 1999.

Celebrating Japan’s extraordinary win over South Africa in Brighton in 2015. Photograph: Tom Jenkins/The Observer

It took him a year to recover but Jones turned himself into one of the world’s great rugby coaches. The lessons of his past have strengthened him and Jones reached two successive Super Rugby finals with the ACT Brumbies, winning the southern hemisphere’s premier regional tournament in 2001. Two years later he guided Australia to the World Cup final, where they held a far better team, England, to the last minute of extra-time – when Jonny Wilkinson’s drop goal settled the game.

Four years later Jones helped South Africa win the 2007 World Cup. For all that he tried to underplay his 13-week consultancy role with the Springboks, two of their leading players told me they would not have won the tournament without Jones. John Smit, the Springboks’ inspirational captain, and Fourie du Preez, their best player, were emphatic that Jones made the difference and ensured their victory.

Smit and Du Preez also believe that, unless England suffer dramatic injuries to their key players in the 2019 World Cup, Jones will win the tournament for the first time as head coach.

Their views are echoed by George Gregan and Rod Kafer, who helped Jones devise a new way of playing rugby with the Brumbies. Gregan was Australia’s captain in the 2003 World Cup and he insists Jones is the best and most empathetic coach in world rugby. His former players in Australia and Japan do not shy away from recalling the way in which he ripped them to shreds when they failed to do their job. But they also stress that Jones made every one of them a better player.

Jones himself is more interested in revealing some of the worst mistakes he made during his 25-year career. He admits he should have resigned as Wallaby coach in 2003 because he was burnt out. But he flogged a limited team and clashed with his fellow coaches and administrators. Jones was sacked in 2005 after Australia lost seven of eight Tests – a painful end which left him in tears but vowing to return to the highest level of international coaching.

He then made the mistake of rushing back into his next job, as head coach of the Queensland Reds, and he hit rock bottom in May 2007. The Reds lost 92-3 to the Bulls in Pretoria. Yet, typically of Jones’s rollercoaster career, he was in the Springbok dressing room four months later as some of the same Bulls players helped South Africa hammer England 36-0 in a World Cup pool game.

A Japan fan pays tribute to Jones and Japan’s win over South Africa. Photograph: Tom Dwyer/Seconds Left/Shutterstock

South Africa gave him three months of respite in a testing five years. From December 2003 to Christmas 2008 it was a grind. “That’s appalling, mate,” he told me, “when you consider how much I love coaching.” His last two years with the Wallabies were bleak. The Reds were even worse. The fun and harmony he experienced with the Springboks was then sucked dry by 14 months of grinding away at Saracens. Jones did not enjoy this spell in English club rugby.

Japan was a backwater of the game but it rescued him when he returned to the country in 2009 to coach Suntory. He eventually became national coach in 2012. Jones transformed Japan as a rugby country and they, in turn, gave him new life as a coach. His impact on Japanese rugby ran deep. Whenever he returns to Japan now he is struck by the latest statistics detailing the number of kids playing rugby. The increase, since Japan beat South Africa in 2015, is between 150 and 200%.

Of course there were difficult days in Japan. He lost his father and also suffered a stroke in October 2013. “People always ask if the stroke changed me,” Jones said in one of our many interviews. “It did. I think I became less intense and learned to relax more. I’m sure the England players will find that amusing. Most of all, it made me grateful to be working again – in a job that gives me such pleasure and satisfaction.”

I was surprised again when Jones said that, after the stroke, he found comfort in going to church in Tokyo every Sunday. “It was a nice coincidence that the pastor was a South African,” Jones said. “He was a big Afrikaans guy who loved rugby. I think he was pleased I turned up.”

Jones is a calmer coach now. He knows he cannot win every battle and needs to pick them wisely. English rugby is scarred by bitter in-fighting between the clubs and the union. If Jones had not learned some hard lessons while in charge of the Wallabies, he could have become swamped in the mess. But, apart from one obvious example when he became engaged in a spat with Bruce Craig, the owner of Bath, he has mostly kept a lid on his frustrations with the English game. He has stuck to his core task of selecting and coaching the national team.

The way in which he has changed the culture of the England team, and discovered the leaders it had been lacking, is eye-opening. In one of his many distinctive insights he suggested Japanese and English society are not that different in terms of the layers that need to be stripped away. In the end he always seems to find a way to the heart of his team.

I share the view of many of his former players that Jones might, just in time, have turned England into the team to beat in this World Cup. I remember him talking about the end of that epic World Cup match between South Africa and Japan in September 2015. His Japanese featherweights had slugged it out with the Springbok heavyweights all afternoon. There was a minute left on the clock when, with South Africa leading 32-29, Japan were awarded a penalty close to the Springbok tryline.

An historic draw was on offer and Jones saw his captain, Michael Leitch, ask the referee how much time was left. “Take the three!” Jones screamed. “Take the three!” Leitch, who was born in New Zealand but had become thoroughly Japanese after 15 years in his adopted country, could not hear him. Jones had spent three years convincing Leitch to throw off the shackles of Japanese subservience to become a dynamic leader. That morning, over coffee, he had told Leitch to trust his instincts.

Jones knew a draw would represent an incredible result. So he was furious when he realised Leitch had rejected the match-saving penalty and was trying for the win. But Leitch had shown bravery and belief. He was prepared to risk defeat to get the win. He had absorbed the lessons Jones had given him. All these years later Jones looked at me and grinned. “I thought: ‘Good on you, mate. Go for it.’”

Japan shocked the world by scoring a last-minute try to seal victory. South Africa 32, Japan 34.

That afternoon in Brighton led directly to Jones becoming England coach and now, four years later, he is back in Japan for another World Cup. Today Jones and his squad are in the coastal resort of Miyazaki preparing for their opening match, against Tonga next Sunday. He will relentlessly badger, urge and encourage them – and hunt down any sign of complacency.

Jones believes strongly that England can become world champions. Soon it will be up to his players to show the same courage and resolve Leitch and Japan did when they made history for themselves. I can still see Jones now, reliving that match-winning moment for Japan, and telling me: “My captain and my team surprised me. But we had prepared them for that very moment and they had the courage to take it. As a coach there is nothing better. I sat back and thought: ‘This is their moment and you did your bit to help them get here.’”

Jones paused and then, eventually, he nodded. “That’s why we do it, mate. That’s what makes it such a beautiful game.”

My Life and Rugby by Eddie Jones, with Donald McRae, will be published on 21 November by Macmillan