I watched without blinking as a row of six hands performed a bullet-fast dance: a splash of soy sauce, a shower of condiments, a high-speed ladle of stock, a flamboyant shake of drained noodles and a final flick of garnish. The gracefully choreographed culinary performance was finished in seconds ­– before a further set of puppet-like hands whisked away the bowls and the dance began once again.

I had to squeeze next to the small restaurant’s cash register in order to see this dance in the cramped, narrow kitchen. It was a humble setting – but it’s one that produces one of Japan’s most revered culinary creations: the nation’s very best ramen.

Served in more than 24,000 restaurants across the country, ramen is increasingly revered for its culinary complexity, from the depth of the broth to the perfect bite of the noodles. It’s the culinary embodiment of the eternal Japanese quest for harmony, beauty and balance. And in December 2015, the dish received the ultimate culinary accolade when Tsuta, a humble backstreet ramen restaurant in Tokyo’s Sugamo district, received a Michelin star.

But the people’s choice of ramen, consistently named Japan’s tastiest in magazines, TV shows and specialist food blogs for more than a decade, is consumed far from the capital, in the less predictable confines of a tiny, family-run noodle shop in Wakayama City ­– a part of Japan that Michelin doesn’t even cover.

Ide Shoten was catapulted from culinary anonymity on 1 January 1998. Out of tens of thousands of ramen restaurants, it was picked as the country’s most delicious during a high-profile New Year’s Day broadcast of the popular television show, TV Champion.

This high-level proclamation paved the way for the restaurant’s soaring popularity (even today, there are 30-minute queues every weekend) and is widely credited with sparking a ramen boom across Wakayama City.

As for the secret to Ide Shoten’s competition-slaying popularity, the list of reasons cited by fans is as long as it is appetite-whetting: the rich fusion of long-stewed pork bones and soy sauce; the fine balance of flavours in the broth; the thin and straight noodles; the deep flavour of the chashu (roast pork).

As a noodle-loving Tokyo resident, it was only a matter of time before I found myself on a pilgrimage to the coastal city, south of Osaka, to taste the holy grail of ramen dishes.

I’m not the first person to head to Wakayama on a near-spiritual mission, and I am unlikely to be the last. Wakayama Prefecture – a heady sprawl of wild mountains, hidden temples and old shrines – has long been famed for the ancient pilgrim trails that sprawl across its rural landscape.

Pilgrimages of the ramen variety, however, are also on the up – as reflected by the fact that Wakayama City’s enterprising tourism office has produced an illustrated ramen map. It’s just as well, because visitors lured by ramen, known locally as chuka-soba (Chinese noodles), are spoilt for choice. There are more than 50 ramen restaurants in the city, which is no mean feat for a place with just 360,000 residents.

I arrived at Wakayama station on an icy-cold morning with big expectations and an empty stomach, four hours after setting off on my noodle odyssey from a pre-sunrise Tokyo. Ramen map or not, Ide Shoten was easy to find: a straight-forward 10-minute stroll from the station, identifiable by a red paper lantern and old yellow-and-red shop sign.

The facade – at first sight, more rundown hardware store than celebrated restaurant – was as low key as its interior. The cosy space, with 10 counter seats surrounding a small eight-seat table in the middle, was imbued with a vintage feel that’s common in local Japanese restaurants: plastic red counters, a retro-looking wall clock, a humming, self-service water machine by the door. Baskets of boiled eggs and clingfilm-wrapped mackerel sushi sat on the tables (a Wakayama ramen tradition), and the walls were covered in messages from famous customers.

At 11:30 am on a nondescript Tuesday morning, the just-opened shop was already humming with customers. Sitting at the central communal table, I was surrounded by suited salarymen, solitary students and Chinese holidaymakers.

Just as my mouth started watering, the owner Norio Ide – a friendly 73-year-old with short grey hair, glasses and a wide smile – rushed to greet me and sat down for a quick chat.

“I hated ramen noodles when I was a child,” he laughed. “My mother used to sell it on a yatai (street cart) and I was embarrassed about this in front of my friends. So I disliked ramen for a long time.”

However, after an eclectic string of jobs – from logistics driver to gas deliveryman – he eventually followed in his mother’s footsteps by taking over her business and opening the restaurant 30 years ago.

Today, fuelled by the awards and resulting media coverage, between 600 and 1,000 bowls of ramen are served daily to customers, 70% of whom come from outside the region, from southern Okinawa to northern Hokkaido and across Asia.

Most people come to sample an iconic bowl of tonkotsu-shoyu (pork bone and soy sauce), one of the two main types of Wakayama ramen (the other has a lighter shoyu (soy sauce) broth).

As I probed Ide about his restaurant’s popularity, he was modestly non-committal about the gold-star techniques that have transformed his humble broth into the stuff of ramen dreams.

“Well, we keep adding to the same broth base every day, building on the flavour, so that makes it very deep,” he said . “Um. That’s it, really. I simply make noodles with my heart.”

As I continued questioning him, he laughed at my use of the word “artform”. “It’s more simple than that,” he said. “We’ve been lucky. I honestly just do the same thing every day.”

The cacophony of satisfied slurping ­ was making me hungry, so I ordered lunch. The menu, written on the wall, could not have been simpler: chuka-soba for 700 yen, with 100-yen options of extra chashu or noodles.

I opted for a no-frills chuka-soba, and in less than a minute, was served a steaming white ceramic bowl of Japan’s finest ramen.

I was immediately struck by the colour: the broth was a rich and earthy orange, and tasted as deep and complex as it looked. It was decidedly rich with strong pork hints (much thicker than the Michelin-starred Tokyo ramen soup), yet managed to be complex without being too heavy.

Firm noodles supported the strong flavour, which was balanced with three thin slivers of chashu, refreshing coils of green onion, a pink fish cake and a palate-cleansing clutch of pickled bamboo shoots.

Slipping into the formal pose of ramen eaters, I tilted my head over the bowl and raised my chopsticks – and the next 10 minutes passed in a blur of sips, slurps and contented sighs as I devoured the delicious concoction.

I was not alone in my appreciation. Shinichi Masuko, 49, a sharply suited salarymen from Osaka sitting at the same table, said, “I’ve been here six times. You could say I’m an addict. It’s the soup I love most. Such a complex flavour.”

Bearing in mind the queue, however, there was no time to chat – and after bidding farewell to Ide, I headed into the drizzling rain with a warm post-ramen glow.

But before getting the train back to Tokyo, there was one last thing to do. I took a short drive to Marina City – a waterside complex with an expansive food market – where I picked up a bag of the region’s famed mikan (mandarin oranges) plus the ultimate souvenir: an Ide Shoten ramen kit, complete with packs of soft noodles and silver pouches of soup broth to replicate the dream dish at home.

Finally, I headed to the complex’s natural hot spring onsen. Here, with a belly full of ramen, I lowered myself into the steaming outdoor bath and soaked up the sea view: perhaps the ultimate way to savour ­– and digest – a pilgrimage to the spiritual heartland of Japan’s ramen world.

If you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.