The clock has only just struck 7am on 2 January, a national holiday, but already the streets of central Tokyo are humming as thousands of people negotiate for position alongside the empty roads.

Near the start, the crowd is 10 deep. University bands and cheerleading squads stomp and yell, the male cheerleaders waving their arms in what looks like some form of exaggerated semaphore. High overhead, helicopters appear and disappear between the cut-glass skyscrapers.

This is the Hakone ekiden; one of the toughest mass-participation endurance events in the world. Run over two days, from central Tokyo to the foot of Mount Fuji and back, the 135-mile distance shared between relay teams of 10 runners, the ekiden brings Japan to a standstill. People who have no other interest in running switch on the television and settle down at home with their new year mochi (rice cakes).

Supporters of the Ritsumeikan university team at the national ekiden in Nagoya (Adharanand Finn)

The race is Japan's biggest annual sporting event, with spectators lining the length of the course, and a huge national TV audience share comparable to that of the Super Bowl in the US and greater than the FA Cup Final often gets.

Long-distance running has a long and noble tradition in the Land of the Rising Sun. As part of the rebuilding process after the devastation of the Second World War, many companies set up sports teams to help raise the morale of their workers. Many of Japan's great marathons, such as the Lake Biwa and Fukuoka races, were first instigated in the late 1940s – long before the running boom in the West had resulted in the birth of races such as the New York and London marathons (in 1970 and 1981, respectively).

To help the runners train for the marathons, the companies competed in relay races – the ekidens. These were modelled on the ancient courier system that saw runners relay messages between Kyoto and Tokyo during Japan's Edo period (1603 to 1868).

The start of day two of the Hakone ekiden (Adharanand Finn)

Most of the marathons and ekidens were sponsored by big newspapers, so they were given pages of detailed coverage, which generated interest and helped people get to know the athletes. Top Japanese runners became so strong that, by the 1960s, the country dominated the global marathon scene. In 1966, 15 of the top 17 times in the world were run by Japanese men.

Today, however, the ekidens have surpassed the marathons in terms of popularity – with the Hakone ekiden the biggest of them all.

I find the first-leg runners down a quiet side-street, pacing back and forth like boxers before a fight, wearing long, shiny jackets against the freezing temperatures. To an outsider, they seem like kids: callow university students, thrown into a cauldron of excitement. Their eyes look around, furtive, nervous. This may well be the biggest moment of their lives. Certainly, unless they go on to compete at the Olympics, it will be the most important race of their careers.

Of course, all this hype isn't for nothing: now they have to show us what they can do. The marshals call the runners to the start. They form two quick lines. Then the starting gun sounds and they streak off, racing down the road. And that's it. They're gone.

The Tokyo university team on winning the 2014 race (Adharanand Finn)

The best way to follow the action is on television. On screen, the runners can be seen bunched up in a large group, running at faster than 60-minute half-marathon pace, for a 21.4km leg (almost exactly half a marathon). Incredibly, all 23 of the runners in the race are going at this pace, which is well inside Japan's national half-marathon record.

By the end they slow down – but the first leg is nevertheless still run so fast that the first three finishers complete it in a time equivalent to a sub-61-minute half-marathon (the world record is 58 minutes 23 seconds). Only four Japanese men in history have managed that time in an actual half-marathon, yet here in Hakone, three students have done it on the first leg. And they haven't even broken the stage record.

The gauntlet is hurled down, and as the second-leg runners head off on their way, they have a lot to live up to. But this is how ekiden – and in particular Hakone – works. The performance of the athletes in each leg inspires subsequent runners to race harder still.

All of the race's 10 stages are close to a half-marathon in distance, and over the two days, an incredible 30 students run a half-marathon equivalent time of under 63 minutes – and that's excluding the times on the race's sixth and fastest stage, which is mostly downhill.

Two Hakone runners are helped by teammates after completing the final leg on day one (Adharanand Finn)

For comparison, among British runners, only three men managed a half-marathon in that time during the whole of 2014.

In race after race, the disparity between UK and Japanese road running is jaw-dropping, especially considering that participation in marathons and half-marathons is pretty much equally as popular and widespread in the UK as it is in Japan. Only six British men have run half-marathons in under 66 minutes so far this year; in Japan, in a single race at the recent Japanese university half-marathon championships, 265 student runners managed it.

It was statistics like these that inspired me to travel to Japan, on a mission to find out just why they were so good. What is behind this incredible depth of running talent?

After six months, the answer was fairly clear. In Japan, road-running and marathons are a serious sport. The top runners are stars. They are paid to join the best company teams, earn a good salary and are able to live well as professional athletes. I estimate there to be around 1,500 professional distance runners in Japan. In the UK, there are about 20.

In fact, the system in Japan is so developed and well supported that I soon realised the really interesting question is not why they are so good, but why they are not even better.

The final leg runner for Komazawa University is inconsolable as his team finishes second in the Hakone ekiden (Adharanand Finn)

When you watch the London Marathon next weekend, you may not know the name of the winner, but you'll hardly be surprised if he is from Kenya or Ethiopia. If he is Japanese, you'll probably fall off your chair. In fact, this year, not a single Japanese elite runner has been entered into the London men's race, reflecting the fact that the country's male marathoners rarely run well outside of Japan.

Japan's female runners have fared better in recent years, with Yukiko Akaba finishing third in London in 2013. In the World Championships of the same year, Japanese women finished third and fourth in the marathon.

But in 2014, 95 of the world's top 100 marathon times were posted by runners from Kenya or Ethiopia. In such a widely participated and accessible sport, that's an incredible level of domination. It's no wonder that most of the Japanese corporate teams, and many of the country's university and high-school teams, have scouted east Africa for runners to sign up to help them win the top ekidens. Yet one of these Kenyan runners living in Japan told me: "Here, they love athletics, it is amazing. More than any other country. More than Kenya. But the training is not good. If they trained like in Kenya, all the world records would come from Japan."

Indeed, in a race where 265 students can run as fast as the very top European runners, why is the winner not running like a Kenyan? Try as they might, in every big global race, the Japanese almost inevitably finish behind the top east Africans. Why?

Waseda University's last-leg runner breaks down as his team finish fourth (Adharanand Finn)

I asked the Kenyan runner what he felt was wrong with the training in Japan, and his answer surprised me. "They train too hard," he said. "Too many long runs." There is a belief, held strongly in Japan – but also shared by many runners in the UK – that to do well you just need to work hard. If your results are not going well, train harder. Feeling tired? Train even harder.

But in running, sometimes you need to train smarter, not harder. In Kenya, for example, the runners are particularly good at resting. If they are feeling tired, it is not unusual for them to skip a session or train slowly. It sounds simple, but it is a trick few runners manage to get right.

But it is not just the training. Another reason the Japanese don't do so well in the big global races is because all of their focus is on ekiden. Almost all the top runners in Japan run for university or corporate teams whose primary goal is to do well in the big ekiden races. This is what the runners are paid for, and their training schedules are arranged to peak for these races. In the ekidens, times are regularly run that would break national records in straight races (they don't count as records, because the runners don't start on a fixed line and the distances of the races are not pegged neatly to set lengths as they are in events such as a 10km or a half-marathon).

Former professional runner Kenji Takao tells me that when he was running, his team won the biggest professional ekiden – the New Year ekiden – six years in a row. Then, in the seventh year, they didn't win. "Nobody ever said anything to me if I ran badly in an individual race," he says. "But when we lost the ekiden I got mail and faxes from angry fans and from my bosses." He laughs. "Only ekiden matters," he says.

Fans wait at the finish of the Hakone ekiden, hours before the runners arrive (Adharanand Finn)

This unquestioning dedication to hard work and the obsession with ekidens result in many of Japan's best runners burning out from physical and mental exhaustion at an early age. With even the high-school ekidens getting wall-to-wall national TV coverage, there is a lot of pressure on the young athletes and also on their coaches to achieve good results.

"Ekiden is destroying our athletes," says Takao, who now coaches one of the top university teams. He adds that coaches often don't think about the long-term prospects of their athletes, particularly at high school and university level. The pressure to win is too great, he says, so they push the athletes too hard.

He is particularly critical of Hakone. He says the preparation for the Hakone ekiden requires regular 30km runs from student runners. "It's too long, too hard."

It's a conundrum. Ekiden makes the sport popular and gives rise to the world of professional, salaried distance running unique to Japan, which in turn provides the support and system to produce so many high-level runners. But ekiden and the continual push to get the most out of the young runners, risking their long-term prospects, is also the source of many problems. With the marathon likely to be the most prized event in Japan come the Tokyo 2020 Olympics, it's a conundrum Japan is going to have to solve soon if it is to have any chance of getting on the rostrum among the east African runners.