Situated on nearly five acres of oceanfront property in Honolulu, Hawaii, the colorful gardens of this stunning, serene estate are hidden behind walls of solid white. From the street, you can't even catch a glimpse. And that was precisely the point.

"I think that Doris came here for privacy. She was, I don't wanna say hounded, but people were certainly interested in her life, and her money," said Leslee Michelsen, the curator of the Shangri La Museum of Islamic Arts, Culture & Design. But before Shangri La was a museum, it was a home, the private residence of an intensely private heiress.

Doris Duke's waterfront home in Honolulu, Hawaii. CBS News

Doris Duke was once known as the "richest girl in the world." Born in 1912, she was the only child of James Buchanan Duke, founder of the American Tobacco Company and benefactor of Duke University. When James died in 1925, Doris was just twelve years old. He left her a massive fortune.

Although Doris did her best to stay out of the limelight, the press documented her every move, including her surprise wedding to James Cromwell in 1935. The couple immediately left the country on an almost year-long trip around the world.

"The kernel of Shangri La was really born on the honeymoon trip," said Michelsen, "and we know that because Doris was very taken with her time in South Asia. She was very taken by the Mughal architecture that she experienced there, especially the Taj Mahal."

Doris Duke at the Taj Mahal in India. CBS News

When the Cromwells finally arrived in Honolulu – intended to be a brief stopover – they decided to stay and build their own Taj Mahal-inspired structure there. Hawaii offered privacy and a relaxed pace of life.

Shangri La was designed by Marion Sims Wyeth, the same architect who designed Mar-a-Lago. Although Duke divorced Cromwell shortly after construction was completed, she spent the rest of her life traveling back and forth to Hawaii, tweaking Shangri La.

Doris Duke and James Cromwell/ CBS News

Michelsen said, "Shangri La's the only house that Doris actually built. She inherited the rest of her homes, and this is the one that she developed according to her own taste and preferences."

The interior of Shangri La is a mis-match of centuries and styles, incorporating elements from Duke's travels throughout the Islamic world.

One of their most celebrated artworks is a mihrab, a rare lusterware prayer niche, made in Iran in 1265. It's just around the corner from a room draped in bright blue fabric made in India in the 1960s.

Duke collected everything from 16th century calligraphy to 18th century jewelry. "Doris wasn't collecting what other people were collecting, even within the boundaries of Islamic art; she was collecting things that nobody else was collecting," said Michelsen.

A 13th century Iranian mihrab, or prayer niche. CBS News

That unique collection forms the basis for what's now the Shangri-La Museum. When Duke died in 1993, she left behind part of her billion-dollar fortune to establish the Doris Duke Foundation for Islamic Art.

Today, tours of Shangri La are sold out months in advance. The museum has started an artist-in-residence program, inviting contemporary Islamic artists to study and work in Duke's former home.

Visitors at the Shangri La Museum of Islamic Art, Culture & Design in Honolulu. CBS News

What was once the most private of retreats is now meant to serve the public. It's one of just a handful of Islamic art museums in the United States.

Michelsen said, "I think, especially in the current socio-political climate, that the arts and cultures of the Islamic world are poorly understood. And I think that that's something that Shangri La can offer to the public. We can be your point of entry to this very rich, wonderful world."

A rich, wonderful world that once captivated the richest girl in the world.