(CNN) For decades, he was known as a godfather of excess. The wealthiest man in the world for many years, the sultan of Brunei knew how to spend the vast riches that flowed from the oil deposits bestowed upon the tiny Southeast Asian nation he controls with absolute power.

But now, the 72-year-old sultan of the former British protectorate is in the spotlight for a very different reason as he comes under fire from governments and activists around the world for introducing the Sharia Penal Code, which includes punishments such as death by stoning for adultery or homosexuality, or amputation of limbs for theft.

Critics say this move towards the most draconian form of Sharia law is a sign of the rising influence of conservative Islam across Southeast Asia -- and an indication that the aging monarch wants to leave a religious legacy that seeks to compensate for his family's controversies.

"The sultan is getting on in years now, and his family hasn't always had the cleanest of reputations," said Matthew Woolfe, the founder of human rights group The Brunei Project.

"Certainly, there are a lot of people talking about the hypocrisy of laws that the sultan and his government are implementing when his family, in the past, could have been seen as being in violation of these laws with some of their antics," said Woolfe. "Some people see it as a way of cleaning up and perhaps creating his legacy."

Brunei's Sultan Hassanal Bolkiah reviews a Chinese honor guard during a welcome ceremony at the Great Hall of the People in Beijing in September 2017.

During a rare public address to the nation last week, Sultan Hassanal Bolkiah made a case for moving towards a more conservative form of Islam, telling his people that he wants to see "Islamic teachings in this country grow stronger and more visible," adding that "this system preserves and guarantees the rights of all the people regardless of their race and faith."

It's a message which seems at odds with the lifestyles the sultan and his brother Prince Jefri Bolkiah are alleged to have lived. During the 1980s and 1990s, they had a reputation for throwing lavish parties, going on multi-million-dollar spending sprees, and running a "harem" at their palace. They owned luxury hotels in London, Paris and New York. Jefri also famously owned a giant luxury yacht he named "Tits" and two smaller vessels called "Nipple 1" and "Nipple 2." Vanity Fair described the brothers as "constant companions in hedonism."

A combination of pictures created in London on March 31, 2019, shows the facades of five of the nine Dorchester Collection hotels owned by Brunei: (top left to right) Hotel Meurice in Paris, The Dorchester in London, The Beverly Hills Hotel in Los Angeles, (bottom left to right) The Hotel Plaza Athenee in Paris and The Hotel Bel-Air in Los Angeles.

"The sultan of Brunei and his brother Prince Jefri were known for their harems, their excesses in terms of purchasing of cars, their sexual exploits," said Bridget Welsh, a Southeast Asia expert and associate professor of political science at John Cabot University. "All of these women were coming into Brunei in the 1980s, so the image of them being the playboys was very prevalent."

Inside the 'harem'

An American woman, Jillian Lauren, published an autobiographical book "Some Girls: My Life in a Harem" in which she alleged that she was recruited to be one of 40 women for Prince Jefri's harem when she was an 18-year-old aspiring actress in New York in the early 1990s.

Lauren said she spent 18 months in the harem on and off -- and was Prince Jefri's "second favorite" mistress for a year.

"He's crazy, and decadent," Lauren told CNN in 2011, referring to Prince Jefri. "At that time, it was attractive to me, his confidence and his charisma and his recklessness."

In 2015, she told a "60 Minutes Australia" documentary, "The Playboy Sultan," that she had sex with Prince Jefri "hundreds" of times and also had a sexual encounter with the sultan himself. She says her book was banned in Brunei.

"I am a witness to the fact that the sultan was drinking, was committing adultery, was not exactly living on the straight and narrow," Lauren told CNN in May 2014.

Lauren said she felt compelled to tell her story when she saw the announcement in 2013 that Brunei was planning to introduce Sharia law.

"It's maybe indicative of the way that a lot of people in power behave," Lauren said. "Which is, it's one rule for them and another rule for the rest of the people, who don't have so much money and don't have so much power."

In response to CNN's request for comment regarding these claims, the Brunei government said in an email, "we vehemently deny the allegations."

Miss California Shannon Marketic, center, reacts as she is crowned Miss USA 1992.

Another alleged recruit to the harem, former Miss USA Shannon Marketic, tried to sue the sultan and Prince Jefri in a US court in 1998, alleging that she was subjected to demeaning treatment and 32 days of virtual imprisonment at the sultan's palace.

Court documents show that Marketic said that 30 to 100 women would be in attendance at the sultan's palace at any one time; some of them understood they were recruited to perform sex acts, but others including Marketic believed they were hired to perform legitimate promotional work for a salary of around $3,000 per day.

Britain's Prince Charles walks with the sultan of Brunei as they attend a high tea at the sultan's palace in Bandar Seri Begawan, Brunei, in 2017.

"Palace staff told Plaintiffs to attend nightly parties and functions at discotheques where the Sultan, Jefri, and their male friends would allegedly select which women they would have sexual intercourse with," according to the court documents.

The brothers denied the allegations, and the sultan said he had never even met Marketic, Brunei state media reported at the time. The US court eventually granted the brothers immunity from legal action, owing to their joint status as a foreign head of state.

A life of excess

At that time, it seemed that the brothers were untouchable.

In 1996, the sultan threw the party to end all parties to mark his 50th birthday. Guests flew in from all over the world for the lavish two-week celebration, which included a banquet dinner for 3,000 people, a special concert by Michael Jackson in a purpose-built stadium, and a polo match with Britain's Prince Charles. The total price tag was an estimated $25 million -- small change for the sultan.

Michael Jackson preforms before an estimated audience of 60,000 in Brunei on July 16, 1996.

The party was the pinnacle of decades of decadence and mind-boggling spending by the royal family. But it also preceded the beginning of a very public and embarrassing falling out between the brothers.

At the time, Prince Jefri was Brunei's finance minister and the chairman of the government-owned Brunei Investment Agency (BIA). But after the Asian financial crisis in 1997, his extravagant tastes were exposed with an audit of the BIA. The sultan accused the prince of embezzling nearly $16 billion from the royal treasury -- which he denied.

Prince Jefri was stripped of his ministerial role and BIA position, and went into self-imposed exile where he tried to fight a legal battle brought by the sultan for the remaining assets of the BIA.

Eventually, the case ended in an out-of-court settlement.

The sultan of Brunei (left) with his brother Prince Jefri Bolkiah, pictured together after a polo match.

In an effort to recoup some of the losses, a debtor auction took place in August 2001, where 10,000 of Prince Jefri's possessions went under the hammer over six days. The lots included a bronze-plated, eight-foot high Trojan horse which went for $1,470, gold-plated Jacuzzis, porcelain flamingo statues, and simulators for a Comanche helicopter, an Airbus jet and a Formula 1 racing car.

Despite the losses the prince allegedly inflicted on the royal coffers, and with declining oil reserves and a faltering economy in Brunei, the sultan still commands a personal fortune of $20 billion, Forbes estimates

The sultan's life is one filled with hyperbole. He was the world's richest man for many years -- before Bill Gates shot past him in the 1990s. His palace -- Istana Nurul Iman -- in the Brunei capital Bandar Seri Begawan, is the biggest private residence in the world , with 1,788 rooms, 257 bathrooms, and air-conditioned stables for 200 polo ponies.

The sultan and his family also own thousands of luxury cars -- including F1 supercars, rare Lamborghinis and around 500 Rolls Royces, the largest private collection of the luxury British auto in the world

Michael Sheehan, a broker who owns and operates the website Ferraris Online, wrote in 2011 about his visit to the compound which housed some of the car collection in an article republished by Gizmodo, describing it as a "massive example of wretched automotive excess." The 2,500 cars Sheehan estimates he saw, included Bentleys, Aston Martins and Ferraris, many of which were "poster children for deferred maintenance," left to rot in the heat and rain.

"Their lifestyle is literally inconceivable for the vast majority of humanity," said Michael Auslin, a fellow in contemporary Asia at the Hoover Institution, at Stanford University. "It's lavish, beyond belief. Take everything you can imagine in the lifestyles of the rich and famous and multiply it."

Britain's Queen Elizabeth II meets the sultan of Brunei at Buckingham Palace in 2017.

The new generation of the Brunei royal family also appears to have taken up the tradition of living in excessive luxury. The sultan has had three wives -- his first cousin, an air hostess, and a TV anchor -- with whom he has a total of 12 children.

Several of the royal offspring have had extravagant wedding ceremonies in Brunei, including Prince Abdul Malik, the third-born son of the sultan, who was married in 2015. Forbes described it as "a royal wedding to beat all royal weddings," with more than a week of celebrations including a main ceremony that saw the couple dressed in "extravagant" matching outfits draped in jewels, with the bride holding a bouquet made of gems and precious metals instead of flowers.

Brunei's Prince Abdul Malik, the second son of Sultan Hassanal Bolkiah, ties the knot with IT instructor Dayangku Raabi'atul 'Adawiyyah Pengiran Haji Bolkiah, 22, in a ceremony in April 2015.

Another of the sultan's sons, Prince Abdul Mateen, is an Instagram star with more than a million followers who watch him play polo and spend time on yachts, private jets and in world-class hotels.

And Prince Azim, the sultan's second-born son, is well known for hosting extravagant parties with celebrity guests including Pamela Anderson, Janet Jackson and Mariah Carey. Notably, the latter two are both winners of the "Ally Award," which honors those who support the LGBT community.

Jared Murillo, Prince Azim of Brunei and Pamela Anderson attend the launch of her exclusive Coco De Mer collection at Morton's on December 5, 2017, in London.

Since Brunei's new Shariah penal code was announced, several A-list celebrities including George Clooney, Elton John and Ellen DeGeneres have started a protest campaign, leading a boycott on luxury hotels around the world owned by the Brunei government.

Insulated from criticism

For most of the estimated 450,000 Bruneians -- 79% of whom are Muslim -- the wealth and spending habits of the royal family is something they have to accept without question.

The sultan is essentially a one-man government -- the "Supreme Executive Authority" with titles including prime minister, defense minister, finance minister, and foreign affairs minister. The media is tightly controlled; Brunei is ranked as one of the worst countries in the world in the annual press freedom index, coming in at 153 out of 180 -- which places it below Russia and Myanmar. Criticism against the family or government is illegal and can result in a jail term for sedition.

Portraits of Brunei's Sultan Hassanal Bolkiah and Queen Saleha are seen beside a slogan (partially cutoff) in Bahasa Melayu that reads "Obedience to Allah, loyalty to the king, love for the country" in Bandar Seri Begawan on October 4, 2017.

Many Bruneians are also willing to turn a blind eye to the excesses inside the palace, experts say, as they are also benefiting from the nation's oil riches. There is no personal income tax, there is subsidized housing, free education and health care, and of course, low fuel and electricity costs. Per capita, Brunei is the fourth wealthiest country in the world, according to Forbes in 2017.

"They're probably fairly well insulated from criticism, as the Bruneians are living better than almost anyone else in Asia," said Michael Auslin of the Hoover Institution. "Here you're talking about oil wealth that has been spread throughout society, the majority are benefiting."

Some observers say that the sultan wants to consolidate that support by increasing his grip on power.

Brunei's Sultan Hassanal Bolkiah (center) returns after blessing to the royal couple Princess Hajah Hafizah Sururul Bolkiah and her groom Pengiran Haji Muhammad Ruzaini (left) at the end of the sitting-in-state on royal dais ceremony at Istana Nurul Iman in Brunei's capital Bandar Seri Begawan on September 23, 2012.

The new Sharia laws are part of a "deeper rationale of controlling the state" in order to "legitimize the exploitation of resources which he and his family have benefited from," said Bridget Welsh, of John Cabot University.

In the sultan's national televised address on Wednesday, he reiterated Brunei's Islamic identity multiple times during the speech.

"As a small nation, we always prioritize our prayers to Allah," the sultan said. "This is driven by our national philosophy of Malay, Islamic, Monarchy (MIB) that has run here since many hundreds of years ago."