As Fern Mallis so aptly put it as she began her latest 92Y “Fashion Icons” conversation last month, if the designer Thom Browne had a theme song, it would have to be the Frank Sinatra mega-hit “My Way.” Since launching his first menswear offerings in 2001, the 52-year-old Pennsylvania native has remained true to his beliefs in unusual proportions, wearing suits without socks, and pairing heavily tailored jackets with shorts.

While acceptance of his ideas—by both customers and retailers—was far from immediate, Browne has become one of the most successful names in fashion, with a business (now both men and women’s) valued at over $100 million, nearly 20 directly operated shops around the world, accounts at close to 300 of the world’s finest department and specialty stores, and a shelf full of awards. Which is particularly remarkable for a man who doesn’t sew or draw (but, as Mallis pointed out, sketches competently), relying instead on his design team (which he calls “the best in the world”) to execute his singular vision.

To say all of this was the fulfillment of a childhood dream would be anything but the truth, Browne told Mallis during their revealing chat. He was the middle of seven children in a conservative, Catholic family, the son of two attorneys, and no one in his home cared that much about clothes. “Fashion was the last thing we ever talked about,” he noted. “My father was a traditional classic businessman; if he didn’t have a suit to put on in the morning, he didn’t know what to wear. As for the kids, if we needed something new, we’d just point to something in the J. Crew or Lands End catalog, and that’s what we got.”

He studied economics at Notre Dame University (where much of his family went and where he could pursue his then-passion for swimming) and worked briefly in financial consulting (including a brief stint at Kohls) after college until he decided he needed to make a fresh start in Los Angeles. He was working in the film industry, as both a production assistant and actor – he had to change his name to Thom (from Tom) to get his SAG card – when he bonded with Johnson Hartig, who went on to start Libertine Clothing.

“I think he was the only other person in L.A. who wore a sportcoat,” Browne told the audience. “Let’s face it, most people in L.A. have no style at all; it’s such a jeans and t-shirt kind of town.” The duo (who remain close) soon spent their times prowling thrift shops and then re-tailoring the vintage clothing into looks that fit their sensibility. “I think that was the beginning of what would become the look of my collections,” says Browne.

Still, it wouldn’t be until four years after he moved to New York City (during which he was a wholesale account manager at Giorgio Armani and a designer at Club Monaco) that Browne had the courage to put his money where his mouth was. Working out of his apartment, Browne had clients come to order custom-made or made-to-measure suits. “I think I sold five the first year,” he said.

Still, he believed enough in his vision that he wore his own suits on a daily basis on the streets of the Big Apple. “Sure, a lot of people looked at me in a funny way, because to their eyes, I was wearing clothes that didn’t fit,” he recalled. “But I didn’t care; in fact, I wanted people to see something totally different. If you’re going to make tailored clothing, I believe you need to push the envelope very far.”

Eventually, the right people took notice, with Paris-based specialty store Colette and New York’s legendary Bergdorf Goodman buying his 2003 collection. “What Bergdorf did was very special; they really were visionaries,” says Browne. “I think we both knew that they could attract a customer who could really appreciate a well-made piece of clothing.”

Despite some blips (including needing to find outside financing more than once), his business has grown exponentially over the past 15 years. Along with a well-regarded sportswear line and a popular women’s line (a favorite of former First Lady Michelle Obama), he has collaborated with Brooks Brothers and Moncler, and speaks proudly of both of those collections. That said, he now claims to be tired of doing collaborations and has no real interest in working on a lower-priced line for a mass-market or fast-fashion retailer.

“Honestly, I don’t know how I could translate my sensibility to those sorts of businesses,” he said frankly.

Still, he’s open to more new opportunities. He’s focusing on growing his online business (he currently sells on Farfetch); he is considering entering the worlds of cosmetics and fragrance (provided he could find the right partner); and he would love to see his past collections—all of which he has archived—exhibited in a museum (which seems entirely possible, since his partner of six years is Andrew Bolton, the head curator of the Metropolitan Museum of Art’s legendary Costume Institute).

Equally important, he noted, “I want to be a little more conscious about the business side of the business. I am always looking for better ways to balance the commercial side with the creative side. But at the end of the day, and this would be my advice for anyone going into fashion, the most important thing is to do work that means something to you and to create something that can live on, not just for a season or two, but for a lifetime or two. Quality is the most important element of any piece of clothing.”