We've flown into Kuala Lumpur with the intention of going jalan-jalan, or on the road to the Malays. I lived in KL, as everyone calls this crazy metropolis, for five years, and it is still like a second home. But this time there is no hanging around, as my wife and I are setting out towards the northern border with Thailand, through the heartland of the Malay peninsula.

This part of the country is often overlooked by tourists, who prefer the golden beaches of the east coast or paradise islands such as Tioman or Langkawi. Our final destination will be Penang, an island my Malaysian friends tell me has suddenly become one of south-east Asia's hottest destinations.

KL is the modern face of Malaysia, a skyscraper city of the future, where Blade Runner meets Bollywood. There are few reminders of the complex history of a country that declared independence from Britain only 55 years ago. Driving out of the centre we soon find ourselves lost in a sprawling urban mass as intimidating as Los Angeles, and it is only with a little luck that we reach the busy north-south highway.

Kuala Lumpur station

The road is surrounded on both sides by rolling hills, marked by geometric lines of palm oil and rubber plantations. The scene resembles an Escher drawing, and there is little trace of the dense rainforest that once covered most of Malaysia. After we've been going an hour, the landscape changes suddenly and dramatically, with massive limestone outcrops leaping hundreds of craggy feet out of the flat plains. Up on a hill, a giant but rather tatty billboard announces that we have arrived at Ipoh – City of Millionaires, and our first stop-off.

Soon we are sitting in the legendary Sinhalese Bar, the sole Sri Lankan bar in Malaysia, whose decor is unchanged since it opened in 1931. I'm sipping a Tiger beer in an iced glass so cold it almost takes the skin off my fingers, and talking to our guide for the next few days, Hong Law Siak, who runs the local heritage association.

"People tend to forget that the modern, developed Malaysia owes its existence to the riches generated by tin and rubber," Hong says. "And in those days Ipoh was as important as Singapore and Kuala Lumpur."

But when the bottom dropped out of the tin market in the 1980s, Ipoh missed out on all the development that so radically altered the face of KL.

"So instead of glitzy, high-rise office blocks and space-age shopping malls, Ipoh has remained pretty much unchanged from the pioneer era. The traditional Chinese shop-houses have been left intact, as have grand colonial landmarks like the Anglo-Moorish railway station, known here as the Taj Mahal of Ipoh."

We leave the bar and wander along the street. Hong points out ornate Chinese clan houses, an abandoned photographer's studio with 1960s black and white prints in the window and a Madras textile shop, where the aged assistants still wear dhotis, scribble sales into a dusty, giant ledger, and an ancient poster on the wall proclaims that their checked sarongs were once Made in British India.

We disappear down the narrow Panglima Lane – known as Concubine Lane when it was lined with gambling and opium dens, brothels and the discreet residences of the many concubines kept by rich Chinese tin tycoons. I'm not surprised when Hong tells me that Ipoh was used as a backdrop for the French film Indochine, starring Catherine Deneuve, about the final days of French colonial rule.

Discovering the eclectic cuisine of Malaysia is always one of the most exciting parts of travelling here, and Ipoh is no exception. Hawker stalls around Pasar Besar, the teeming wet market, specialise in kway teow – wok-fried rice noodles with egg, prawns and juicy cockles – pork satay, paper-thin poh piah spring rolls stuffed with bean sprouts, and claypot chicken. Indian restaurants offer a dozen different curries around a mountain of rice on a banana leaf. But the best meal we eat here is Malay nasi kandar, a self-service feast of at least 40 dishes, where a plate heaped with delicious okra and bitter gourd, spicy beef rendang, squid and fish curry costs less than two pounds.

The next day, Hong turns up in his car to drive us around the surrounding Kinta valley, where there used to be 1,000 tin mines. These are now deep, manmade lakes. Today, virtually none of the mines is still functioning: there is still tin beneath the ground, but extracting it is too expensive.

Assam lakha, hot and sour soup, Penang

The century-old mining town of Papan is certainly no Klondike boom town: most of the shop-houses that line the main street are abandoned or dilapidated. People pass the time playing mah jong or talking tin prices in the Lee Seow Yoon coffee shop. Papan is Hong's special project: he has lovingly recreated the dispensary where Sybil Kathigasu, known as the Malaysian Florence Nightingale, secretly treated guerrilla partisans during the Japanese occupation in the second world war.

The war theme continues down the road at Batu Gajah, whose eerie Anglican cemetery is named God's Little Acre. Here stand the graves of three British planters whose violent deaths in 1948 sparked the Malayan Emergency – 12 years of fierce fighting between Chinese communists, British forces and, later, the newly independent Malaysian government. The inscriptions are chilling: "died at the hands of terrorists", "murdered by terrorists".

The mood lightens when Hong unveils his next surprise, a quite amazing ruined castle, straight out of a Somerset Maugham story. I had heard about Kellie's Castle many years before, but it used to be inaccessible, hidden in the jungle. Now this grandiose folly has become something of a tourist attraction, sitting majestically atop a grassy hillock.

William Kellie Smith arrived in Malaya in 1890, aged 20, and after rapidly making a fortune as a planter he decided to build a romantic castle for his wife – his childhood sweetheart, Agnes. Kellie designed it in ornate Moorish revival style, and planned a rooftop terrace for parties, a basement wine cellar, even a lift to get to the top of the tower. But he died before it was completed. The castle was never inhabited, and after years of abandon it is now overgrown with vegetation and the gnarled roots of towering banyan trees, occupied by monkeys. This being Malaysia, it is apparently also home to numerous ghosts.

We finish the day 10 miles south in the town of Tanjung Tualang, whose enterprising inhabitants had the original idea of transforming their mining lakes into farms for freshwater prawns. Every single shophouse here houses a noisy seafood restaurant, drawing busloads of gourmets each weekend. The succulent prawns are enormous, and steamed to perfection in Chinese rice wine, egg white and ginger. This is pricy for Malaysia, though – a kilo will set diners back £15.

Next day we head north from Ipoh, following winding backroads through shady rubber plantations to Kuala Kangsar, which looks at first like a quiet provincial town in a bend on the Perak river. In fact, this is a royal capital, home to the Sultan of Perak, some of the grandest palaces and mosques in Malaysia, and the prestigious Malay College, the nation's answer to Eton.

The Sultan, whose predecessors have ruled Perak since the 1500s, lives in a grand pastel art deco palace, the Istana Iskandariah, but visitors can only get a peek from afar. The far more beguiling Istana Kenangan is a magnificent example of Malay architecture – all wood, but without a single nail. This is open to the public, because it houses the Royal Museum.

But nothing compares to the first view of the Ubudiah mosque, a swirling vision of marble turrets and golden cupolas. It was designed in 1917 by Arthur Benison Hubback, an unsung British government architect whose idiosyncratic Indo-Saracen buildings still stand out in most of Malaysia's major towns. Walking round its cool arcades I spot a young Malay couple, resplendent in pale blue wedding robes, posing shyly for photos. Malay hospitality is so generous (and foreign visitors to Kuala Kangsar so rare) that they even invite us to the reception.

Before reaching Penang, we plan a break in Malaysia's oldest hill station, one of the cool mountain resorts where you can escape the 30C temperatures and 95% humidity that is the daily climate throughout the year. The road signs may say Bukit Larut, but everyone still uses the colonial name of Maxwell Hill.

Most of the hill stations have been modernised, but not Maxwell Hill. The one road up to the 1,250m summit is a rough path with 72 hairpin bends, which only the official government jeep is allowed to tackle. Every hotel development here has ended up closed or abandoned, and although there are a couple of ramshackle bungalows and nature hostels, I have been warned to prepare for minimal facilities.

The reward, though, is a solitary paradise for birdwatchers, butterfly enthusiasts and jungle trekkers, with amazing views as far as the Andaman Sea. We are not to be rewarded today, though: usually in Malaysia, everyone's favourite catchword is boleh – can do – but at the jeep office, where a "ticket sold out today" sign foils our plans, and no amount of persuasion, or bribery, can get us a seat. So we head back to the road, and after a couple of hours are sailing over the eight-mile-long Penang bridge, which links the mainland to the island known as the pearl of the orient.

Cheong Fatt Tze Mansion, Penang

I must admit I did not really believe all the buzz about Penang, but the moment we arrive at our designer B&B it is obvious that the island's capital, George Town, is no longer the rundown place I remember. Unesco world heritage status (granted in 2008) has saved historic buildings from the wrecking ball and bulldozer. Instead of making way for high-rise office blocks and shopping centres, the sumptuous Chinese mansions and maze-like shop-houses are being elegantly transformed into boutique hotels, art galleries and boutiques, funky restaurants and bars. And down in the street, there are still the teeming markets, artisan workshops, strange medicinal shops and delicious 24-hour hawker stalls that make George Town one of the last surviving authentic Chinatowns.

We're staying in Cheong Fatt Tze mansion – known as the Blue Mansion because of its distinctive painted walls – which was built in the 1880s by a Chinese entrepreneur dubbed the "Rockefeller of the East". I actually wandered in here years ago when the house was falling apart and inhabited by squatters. Now it is a symbol of George Town's transformation, restored to its former glory, guestrooms furnished with antiques from mainland China alongside art nouveau treasures over from Europe. Our room, in the old kitchen, has the original hearth and cooker.

For an even better glimpse of Penang's opulent past, there is the nearby Peranakan Museum (pinangperanakanmansion.com.my), a perfectly preserved private mansion filled with treasures.

Wandering through Penang's melting pot of mosques, Hindu temples and incense-filled Chinese shrines, we stumble on Chinahouse (chinahouse.com.my), an eclectic cultural centre that exhibits art-house installations, promotes reggae and soul concerts and showcases gourmet Pacific Rim cuisine – unheard of in the old Penang. Studio Howard (studiohoward.com) is a cutting-edge photo gallery, while Campbell House (campbellhousepenang.com) is a hip guesthouse with a genuine Venetian restaurant – it's run by a chef from La Serenissima who has come to Penang to make his name.

This is just the tip of the iceberg of the metamorphosis of the pearl of the orient. Responsible heritage tourism could well have the same effect on the rest of Malaysia's heartland.