Wisps of frankincense smoke wafted through the bazaar as I plunged through the crowded, labyrinthine passageways of Muscat’s Mutrah Souq. This alluringly musky scent permeates Omani cities and culture, and I was never far from the distinct, earthen aroma floating through the air.

I was lured by the hypnotic perfume curling up from frankincense smouldering in pots outside stores dripping with gold jewellery and silver censers. Tiny open-air shops teemed with spices, mounds of myrrh and piles of dates. Women in full-length black abaya cloaks perused colourful-as-Crayola silk scarves and shawls, while men in white ankle-length dishdasha robes and beautifully embroidered kuma caps inspected heaps of pebble-sized amber-, caramel- and cream-coloured frankincense nuggets.

You may also be interested in:

• A hidden village carved into a cliff

• The land covered in sacred gold

• The world’s first Christian country?

This was Muscat at its most magical, conjuring images from the Bible. The Mutrah Souq was also, surely, one of few places in the world where I could buy gold, myrrh and frankincense – the three gifts presented to baby Jesus by the Three Magi in Christian tradition – under one roof. These were among the most precious gifts imaginable two millennia ago, when frankincense was worth its weight in gold.

Used for 6,000 years as a perfume and panacea, frankincense (from the Old French ‘franc encens’, meaning ‘pure incense’) is an aromatic resin harvested from trees of the rugged Boswellia genus, which grows exclusively in a narrow climate belt from the Horn of Africa to India and parts of southern China. Most of the world’s supply comes from Somalia, Eritrea and Yemen – countries plagued by conflict in recent years, which has negatively affected their frankincense production. But peaceful Oman produces the world’s finest – and most expensive – frankincense, a substance ancient Egyptians called the ‘Sweat of the Gods’.

We burn frankincense to repel snakes

The hardy Boswellia sacra tree thrives in the inhospitable terrain of Oman’s southern province of Dhofar. The value of frankincense resin is determined by its colour, clump size and oil concentration. The most valuable grade, known as hojari, comes from a narrow, dry microclimate belt of the Dhofar Mountains just beyond reach of the summer monsoon that blankets the tip of the Arabian Peninsula in mist.

Today, the frankincense trees studding this region, and a number of caravan routes and ports dating from the 4th Century BC, are part of Oman’s Unesco-inscribed Land of Frankincense World Heritage site. According to Unesco’s description, “the trade in frankincense that flourished in this region for many centuries [was] one of the most important trading activities of the ancient and medieval world.”

From here, caravans with thousands of camels and slaves laden with frankincense set off on a daunting 2,000km-long overland journey across the Arabian Desert – destined for the Egyptian, Babylonian, Greek and Roman Empires – while ships laden with the resin travelled as far as China. Roman polymath Pliny the Elder (23-79AD) wrote that the trade had made the southern Arabians “the richest people on Earth”.

The aspirin, penicillin and Viagra of its day, frankincense was considered an effective remedy for everything from haemorrhoids to menstrual pains and melanoma. Greek military physician Pedanius Dioscorides described frankincense as a cure-all wonder drug, writing that the gummy resin could ‘fill up the hollowness of ulcers’ or ‘glue together bloody wounds’. The Ebers Papyrus, ancient Egypt’s most important document of medical knowledge, mentions frankincense as a treatment for asthma, bleeding, throat infections and vomiting, among other things. The Egyptians imported massive amounts for use also as a perfume, insect and vermin repellent, and to mask the rotting odour when embalming bodies. Frankincense ointment was even found in King Tutankhamun’s tomb when it was opened in 1922.

Burned for purification, frankincense suggested divinity. Its swirling smoke was believed to ascend directly to heaven. Many temples of the ancient world would have been steeped with its unambiguous olfactory stamp.

“We burn frankincense to repel snakes,” said local guide Amur bin Hamad al-Hosni as he led me around northern Oman’s Ad Dakhiliyah region to the 17th-Century Nizwa Fort, whose strategic location at the crossroads of several trade routes led the territory to once be known as ‘The Pearl of Islam’. “And to ward off djinn – evil spirits,” added Maitha Al-Zahraa Nasser Al Hosni, sales clerk in the fort’s gift shop. The shop teemed with a profusion of frankincense essential oils, perfumes and lotions. Bin Hamad al-Hosni’s gold-edged robe was steeped in frankincense fragrance.

During my time in Oman, I was amazed to learn how much frankincense remains an integral part of Omani culture.

In the city of Nizwa, I saw Omanis chewing the edible resin like gum to freshen their breath. “Pregnant women also nibble it,” a storekeeper told me, as expecting mothers believe that its properties will ensure the birth of an intelligent baby. It’s also used in medicines and teas to promote healthy digestion and skin. Omanis perfume their homes to repel mosquitoes, and passing around an incense burner after a meal is considered a mark of hospitality. The type of frankincense used is viewed as both a status symbol and mark of respect.

“Omanis swear that white frankincense ‘tears’ – [clumps of sap] – from Jabal Samhan or Hasik is best,” said Trygve Harris, owner of Enfleurage, one of Oman’s boutique frankincense distillers, and referring to the frankincense harvested along Oman’s south-eastern Dhofar coast. “It’s considered purer, with the most prized aroma. My favourite is oozy black frankincense from the cliffs of Al Fazayah, west of Salalah,” she added. She showed me a room full of old copper alembics in which she coaxes complex and delicate essential oil from the resin. She held out a wicker basket containing small cream-coloured resin tears, and dark brown nuggets resembling large wads of congealed glue. Different soils, microclimates and even times of harvest produce distinct resin colours, and generally, the whiter the resin, the more valuable it is.

In summer, Harris even makes frankincense gelato, which is quickly scooped up by locals at a stall she rents in the Muscat Souq.

Harris first came to Oman in 2006 to source frankincense for her aromatic essential oil store in New York. “But even in Oman, I could only get Somalia oil, not higher-quality Omani oil. Nobody was distilling it for sale back then. Not even Amouage!” she recalled, speaking of Oman’s top perfume company, which specialises in producing luxury frankincense fragrances (a 100ml bottle of Amouage perfume with frankincense base notes costs £283).

In 2011, she relocated to Salalah, Dhofar’s capital, and set up Enfleurage. Today based in Muscat, Harris sells to small international perfume houses, essential oil companies and “people who want boutique, high-quality frankincense and who don’t need 1,000kg”. She produces 2kg to 3kg a day of Boswellia sacra essence, worth £555 per kg.

Virtually all Omani frankincense is harvested from Boswellia sacra trees that grow wild in the searing Dhofar desert, and are owned by the local tribes. The harvest begins each April as rising temperatures cause the sap flow more easily. Workers cut small incisions into the bark, causing it to ooze a white, milky sap that dribbles down the tree like candle wax. The sap is left for 10 days to solidify into a gum. After the ‘tears’ are scraped off, farmers cut the same spot again. They repeat this process several times, with the final autumnal harvest producing the palest and most valuable resin. After about five years of tapping, the tree is not touched for the next five years.

However, in recent years Oman’s rare Boswellia sacra trees have been threatened by rocketing global demand. “Renewed interest for frankincense in essential oils and holistic medicine in the international market has added to the stress on the natural Boswellia habitat," said botanist Joshua Eslamieh, author of Cultivation of Boswellia: Sacred Trees of Frankincense.

Boswellia sacra is now listed on the international Red List of Threatened Species as ‘Near Threatened’, and a recent study published in Nature claims the tree is dying off so quickly that frankincense production will diminish by 50% in the next 20 years. Another report warns that production has already fallen from an average of 10kg per tree to only 3.3kg, and found that the population of Boswellia sacra trees in Dhofar’s Jabal Samhan Nature Reserve has declined 85% in the past two decades.

Omanis swear that white frankincense ‘tears’ from Jabal Samhan or Hasik is best

Scientists have pinpointed drought, overgrazing, insect attacks and illicit harvesting by armed Somali smugglers as reasons for the resin’s decline – which has led the Sultan of Oman to employ armed guards to protect the valleys in recent years.

Yet according to the Environmental Society of Oman’s project manager, Dr Mohsin Al Amri, unsustainable harvesting is the plant’s biggest threat. “Less experienced, part-time workers are damaging the trees by forgoing traditional harvesting techniques,” he said. Smaller, immature trees are being tapped, mature trees are being overharvested to meet soaring demand and fewer Boswellia seedlings are surviving to become saplings and mature trees.

I would never have known it, watching the tide of Omanis inspecting mountains of frankincense nuggets in Mutrah Souq, sorted and priced according to colour and source.

The bazaar’s warren of alleys eventually led me to spill onto Muscat’s harbourfront Mutrah Corniche promenade, curving along the shoreline of the Arabian Sea. It was dominated, exquisitely, by the Mutrah Central Mosque’s turquoise dome and minaret. Traditional dhows bobbed in a bay wrapped by mountains coloured burnt henna. The ululations of imams called the faithful to prayer, and the air was redolent with the unmistakable fragrance of Oman’s signature scent.

Join more than three million BBC Travel fans by liking us on Facebook, or follow us on Twitter and Instagram.

If you liked this story, sign up for the weekly bbc.com features newsletter called "The Essential List". A handpicked selection of stories from BBC Future, Culture, Worklife and Travel, delivered to your inbox every Friday.