Let’s start with a couple of stories from the world of sports. This first one, you probably know … The boy’s father could tell something was different. At seven months, he gave his son a putter to fool around with, and the boy dragged it everywhere he went in his little circular baby walker. At 10 months, he climbed down from his high chair, trundled over to a golf club that had been cut down to size for him, and imitated the swing he had been watching in the garage. Because the father couldn’t yet talk with his son, he drew pictures to show the boy how to place his hands on the club.

At two, he went on US television and used a club that was tall enough to reach his shoulder to drive a ball past an admiring Bob Hope. That same year, he entered his first tournament, and won the 10-and-under division. There was no time to waste. By three, the boy was learning how to play out of a sandtrap, and his father was mapping out his destiny. He knew his son had been chosen for this, and that it was his duty to guide him. He started prepping his three-year-old to handle the inevitable media attention that would come. He quizzed the boy, playing the role of reporter, teaching him how to give curt answers, never to offer more than precisely what was asked.

When the boy was four, his father could drop him off at a golf course at nine in the morning and pick him up eight hours later, sometimes with the money he had won from those foolish enough to doubt him.

At eight, the son beat his father for the first time. The father didn’t mind, because he was convinced that his boy was singularly talented, and that he was uniquely equipped to help him. He had been an outstanding athlete himself, and against enormous odds. He played baseball in college when he was the only black player in the entire conference. He understood people, and discipline; a sociology major, he served in Vietnam as a member of the army’s elite Green Berets, and later taught psychological warfare to future officers. He knew he hadn’t done his best with three kids from a previous marriage, but now he could see that he had been given a second chance to do the right thing with number four. And it was all going according to plan.

The boy was already famous by the time he reached Stanford University, and soon his father opened up about his importance. His son would have a larger impact than Nelson Mandela, than Gandhi, than Buddha, he insisted. “He has a larger forum than any of them,” he said. “He’s the bridge between the east and the west. There is no limit because he has the guidance. I don’t know yet exactly what form this will take. But he is the Chosen One.”

This second story you also probably know. This boy’s mother was a coach, but she never coached him. He would kick a ball around with her when he learned to walk. As a child, he played squash with his father on Sundays. He dabbled in skiing, wrestling, swimming and skateboarding. He played basketball, handball, tennis, table tennis and badminton over his neighbour’s fence, and soccer at school.

His parents had no particular athletic aspirations for him. They encouraged him to try a wide array of sports. He didn’t much mind what sport he was playing, so long as it included a ball. Though his mother taught tennis, she decided against working with him. “He would have just upset me anyway,” she said. “He tried out every strange stroke and certainly never returned a ball normally. That is simply no fun for a mother.” Rather than pushy, his parents were, if anything, “pully”, a Sports Illustrated writer would later observe. Nearing his teens, the boy began to gravitate more toward tennis, and “if they nudged him at all, it was to stop taking tennis so seriously”.

Roger Federer in 1996, aged 15, competing at the World Youth Cup in Zurich. Photograph: Keystone

As a teenager, he was good enough to warrant an interview with the local newspaper. His mother was appalled to read that, when asked what he would buy with a hypothetical first prize money from playing tennis, her son answered “a Mercedes”. She was relieved when the reporter let her listen to a recording of the interview and they realised there had been a mistake: the boy had said “mehr CDs” in Swiss-German. He simply wanted “more CDs”.

The boy was competitive, no doubt. But when his tennis instructors decided to move him up to a group with older players, he asked to move back so he could stay with his friends. After all, part of the fun was hanging around after his lessons to gab about music, or pro wrestling, or soccer.

By the time he finally gave up other sports to focus on tennis, other kids had long since been working with strength coaches, sports psychologists and nutritionists. But it didn’t seem to hamper his development in the long run. In his mid-30s, an age by which even legendary tennis players are typically retired, he would still be ranked world No 1.

In 2006, Tiger Woods and Roger Federer met for the first time, when both were at the apex of their powers. Woods flew in on his private jet to watch the final of the US Open tennis tournament. It made Federer especially nervous, but he still won, for the third year in a row. Woods joined him in the locker room for a champagne celebration. “I’ve never spoken with anybody who was so familiar with the feeling of being invincible,” Federer would later say. They quickly became friends.

Still, the contrast between them was not lost on Federer. “His story is completely different from mine,” he told a biographer in 2006. “Even as a kid, his goal was to break the record for winning the most majors. I was just dreaming of just once meeting Boris Becker, or being able to play at Wimbledon some time.”

It seems pretty unusual for a child with “pully” parents, and who first took his sport lightly, to grow into a man who dominates it like no one before him. Unlike Tiger, thousands of kids, at least, had a head start on Roger. Tiger’s incredible upbringing has been at the heart of a batch of bestselling books on the development of expertise, one of which was a parenting manual written by Tiger’s father, Earl. Tiger was not merely playing golf. He was engaging in what is often described as “deliberate practice”, the only kind that counts in the oft-quoted “10,000 hours rule” of expertise, which states that the number of accumulated hours of highly specialised training is the sole factor in skill development, no matter the domain.

Deliberate practice, according to the study of 30 violinists that spawned the rule, occurs when learners are “given explicit instructions about the best method,” individually supervised by an instructor, supplied with “immediate informative feedback and knowledge of the results of their performance” and “repeatedly perform the same or similar tasks”. Reams of work on expertise-development shows that elite athletes spend more time in highly technical, deliberate practice each week than those who plateau at lower levels.

Tiger Woods practising his swing at 11 months old. Photograph: David Strick/Redux/Eyevine

Tiger has come to symbolise the idea that the quantity of deliberate practice determines success – and its corollary, that the practice must start as early as possible. But the push to focus early and narrowly extends well beyond sports. We are often taught that the more competitive and complicated the world gets, the more specialised we all must become (and the earlier we must start). Our best-known icons of success, such as Facebook CEO Mark Zuckerberg, are lauded for their precocity and their headstarts. Across different fields, it has become more and more common to exalt increasingly narrow focus. Oncologists no longer specialise in cancer, but rather in cancer related to a single organ – and the trend advances each year. Surgeon and writer Atul Gawande pointed out that when doctors joke about left-ear surgeons, “we have to check to be sure they don’t exist”.

In the 10,000-hours-themed bestseller Bounce, UK journalist Matthew Syed suggested that the British government was failing for a lack of following the Tiger Woods path of unwavering specialisation. Moving high-ranking government officials between departments, he wrote, “is no less absurd than rotating Tiger Woods from golf to baseball to football to hockey”.

But Great Britain’s massive success at the 2012 Olympics, after decades of middling performances, was bolstered by programmes set up specifically to recruit adults to try new sports and to create a pipeline for late developers – “slow bakers”, as one of the officials behind the programme described them. It turns out that the idea of an athlete, even one who wants to become elite, following a Federer path and trying out different sports is not so absurd.

Elite athletes at the peak of their abilities do spend more time on deliberate practice than their near-elite peers. But scientists have found that, at a younger age, those who go on to become elite athletes typically devote less time to deliberate practice in the activity in which they will eventually become experts. Instead, they undergo what researchers call a “sampling period”. They play a variety of sports, usually in an unstructured or lightly structured environment; they gain a range of physical proficiencies from which they can draw; they learn about their own abilities and proclivities; and only later do they focus in on one area. The title of one study of athletes in individual sports proclaimed “late specialisation” as “the key to success”; another was titled Making It to the Top in Team Sports: Start Later, Intensify, and Be Determined.

When I began to write about these studies a few years ago, I was met with thoughtful criticism, but also denial. Maybe in some other sport, fans would often say, but that’s not true of our sport. The community of the world’s most popular sport, football, was the loudest. And then, as if on cue, in late 2014 a team of German scientists published a study showing that members of their national team, which had just won the World Cup, were typically late specialisers who didn’t play more organised soccer than amateur-league players until age 22 or later. They spent more of their childhood and adolescence playing non-organised football and other sports. Another football study published two years later matched players for skill at age 11 and tracked them for two years. Those who participated in more sports and non-organised football, “but not more organised soccer practice/training”, improved more by age 13. Findings like these have been echoed in a huge array of sports, from hockey to volleyball.

The professed necessity of hyperspecialisation forms the core of a vast, successful and sometimes well-meaning marketing machine, in sport and beyond. In reality, the Roger Federer path to sports stardom is far more prevalent than the Tiger Woods path, but those athletes’ stories are much more quietly told, if they are told at all. Some of their names you know, but their backgrounds you probably don’t.

In the 2018 Super Bowl, for instance, a quarterback who had been drafted into professional baseball before football (Tom Brady), faced off against one who participated in football, basketball, baseball and karate, and had chosen between college basketball and football (Nick Foles). Later that very same month, Czech athlete Ester Ledecká became the first woman ever to win gold in two different sports (skiing and snowboarding) at the same Winter Olympics.

When she was younger, Ledecká participated in multiple sports (she still plays beach volleyball and windsurfs), focused on school and never rushed to be No 1 in teenage competition categories. Just after her feat, Ukrainian boxer Vasyl Lomachenko set a record for the fewest fights needed to win world titles in three different weight classes. Lomachenko, who took four years off boxing as a kid to learn traditional Ukrainian dance, reflected: “I was doing so many different sports as a young boy – gymnastics, basketball, football, tennis – and I think, ultimately, everything came together with all those different kinds of sports to enhance my footwork.”

In 2014, I included some of the findings about late specialisation in sports in the afterword of my first book, The Sports Gene. The following year, I got an invitation to talk about that research before an unlikely audience –not athletes or coaches, but military veterans. In preparation, I looked through scientific journals for work on specialisation and career-swerving outside of the sports world – and I was struck by what I found.

One study showed that early-career specialisers earned more than others after college, but that later specialisers made up for the headstart by finding work that better suited their skills and personalities. I found many studies showing how technology inventors had benefited from proactively sacrificing a modicum of depth for breadth as their careers progressed. There was a nearly identical finding in a study of artistic creators.

I also began to realise that some of the people whose work I deeply admired from afar – from Duke Ellington (who shunned music lessons to focus on drawing and baseball as a kid) to Maryam Mirzakhani (who dreamed of becoming a novelist and instead became the first woman to win math’s most famous prize, the Fields Medal) – seemed to have more Federer than Woods in their development stories.

In my talk to the small group of military veterans, I mostly stuck to sports. I touched on the other findings only briefly, but the audience seized on that material. All were late specialisers or career-changers, and as they filed up one after another to introduce themselves after the talk, I could tell that all were at least moderately concerned, and some were borderline ashamed of it.

The original one-man band … Duke Ellington. Photograph: Bettmann Archive

They had been brought together by the Pat Tillman Foundation, which, in the spirit of the late NFL player who left a professional football career to become an army ranger, provides scholarships to veterans, active-duty military, and military spouses who are undergoing career changes or going back to school. They were all scholarship recipients, former paratroopers and translators who were becoming teachers, scientists, engineers, and entrepreneurs.

They brimmed with enthusiasm, but they also seemed a little anxious about the fact that their LinkedIn profiles didn’t show the linear progression toward a particular career they had been told employers wanted. They were nervous about starting graduate school alongside younger (sometimes much younger) students, or changing lanes later than their peers. Although they had been busy accumulating unique life and leadership experiences, somehow, in their minds, this advantage had morphed into a liability.

Over time, as I delved further into studies about learning and specialisation, I came across more and more evidence that it takes time to develop personal and professional range – and that there are benefits to doing so. I discovered research showing that highly credentialed experts can become so narrow-minded that they actually get worse with experience, even while becoming more confident (a dangerous combination). And I was stunned when cognitive psychologists I spoke with led me to an enormous and too-often ignored body of work demonstrating that learning itself is best done slowly to accumulate lasting knowledge, even when that means performing poorly on tests of immediate progress. That is, the most effective learning looks inefficient – it looks like falling behind.

Starting something new in middle age might look that way, too. Mark Zuckerberg famously noted that “young people are just smarter”. And yet a tech founder who is 50 years old is nearly twice as likely to start a blockbuster company as one who is 30, and the 30-year-old has a better shot than someone of 20. Researchers at Northwestern University, MIT and the US Census Bureau studied new tech companies and showed that among the fastest-growing startups, the average age of a founder was 45 when the company was launched.

Zuckerberg was 22 when he made his pronouncement about the superior intelligence of young people. It was in his interest to broadcast that message, just as it is in the interest of people who run youth sports leagues to claim that year-round devotion to one activity is necessary for success, never mind the evidence to the contrary. But the drive to specialise goes beyond that. It infects not just individuals, but entire systems, as each specialised group sees a smaller and smaller part of a large puzzle.

One revelation in the aftermath of the 2008 global financial crisis was the degree of segregation within big banks. Legions of specialised groups optimising risk for their own tiny pieces of the big picture created a catastrophic whole. To make matters worse, responses to the crisis betrayed a dizzying degree of specialisation-induced perversity. A US federal government programme launched in 2009 incentivised banks to lower monthly mortgage payments for homeowners who were struggling but still able to make partial payments.

A nice idea, but here’s how it worked out in practice: a bank arm that specialised in mortgage-lending started the homeowner on lower payments; an arm of the same bank that specialised in foreclosures then noticed that the homeowner was suddenly paying less, declared them in default, and seized the home. “No one imagined silos like that inside banks,” a government adviser said later. Overspecialisation can lead to collective tragedy even when every individual separately takes the most reasonable course of action.

Specialised healthcare professionals have developed their own versions of the “if all you have is a hammer, everything looks like a nail” problem. Interventional cardiologists have grown so used to treating chest pain with stents – metal tubes that pry open blood vessels – that they do so reflexively even in cases where voluminous research has proven that they are inappropriate or dangerous. A recent study found that cardiac patients were less likely to die if they were admitted during a national cardiology meeting, when thousands of cardiologists were away; the researchers suggested it could be because common treatments of dubious effect were less likely to be performed.

Arturo Casadevall, an internationally renowned scientist, believes that increasing specialisation has created a “system of parallel trenches” in the quest for innovation. Everyone is digging deeper into their own trench and rarely standing up to look in the next trench over, even if the solution to their problem happens to reside there. Casadevall is taking it upon himself to attempt to despecialise the training of future researchers; he hopes that eventually it will spread to training in every field. He profited immensely from cultivating range in his own life, even as he was pushed to specialise. And now he is broadening his purview again, designing a training programme in an attempt to give others a chance to deviate from the Tiger path. “This may be the most important thing I will ever do in my life,” he told me.

Learning about the advantages of breadth and delayed specialisation has changed the way I see myself and the world. The research pertains to every stage of life, from the development of children in maths, music and sports, to students fresh out of college trying to find their way, to midcareer professionals in need of a change and would-be retirees looking for a new vocation after moving on from their previous one.

The challenge we all face is how to maintain the benefits of breadth, diverse experience, interdisciplinary thinking and delayed concentration in a world that increasingly incentivises or even demands hyperspecialisation. While it is true that there are areas that require individuals with Tiger’s precocity and clarity of purpose, as complexity increases – as technology spins the world into vaster webs of interconnected systems in which each individual only sees a small part – we also need more Rogers: people who start broad and embrace diverse experiences and perspectives while they progress. People with range.

Adapted from Range: How Generalists Triumph in a Specialised World by David Epstein, published by Macmillan and available at guardianbookshop.com

• Follow the Long Read on Twitter at @gdnlongread, and sign up to the long read weekly email here.