New York (CNN Business) When Chip Wilson went to his first yoga class in 1998 at the age of 42, he noticed that people looked uncomfortable sweating in baggy clothes.

"At that time, everyone wore their worst clothing to the gym," the controversial Lululemon ( LULU ) founder told CNN Business' Vanessa Yurkevich in an interview.

Wilson grew up in Canada and started Westbeach, a surf, skate, and snowboard clothing company. He had experience making clothing for functional activities, and saw an opportunity to move into yoga.

"Yoga is about sweating," he said. "As the cotton got wet, it started to bind. On top of that, the instructor couldn't really align the body very well."

Wilson played around with his idea and started thinking about a name for his new brand.

He remembered a time when he was doing business in Japan and told his buyers he would be closing the skateboarding division Westbeach -- called Homless. To his surprise the Japanese bought the name Homless for a "ridiculous price."

"I recognized that the Japanese consumer liked a name with an L in it because the letter isn't in the Japanese language. It sounded American," he said. "So I thought if I ever have another company I am going to put three L's in it." Lululemon was born.

'Super girls'

Wilson tested his new yoga pant design in focus groups. The pants used more Lycra, a stretchy fabric, than other companies had relied on.

"I suspected it would be good, because no one had ever come up with a synthetic fabric that felt like cotton," he said.

Lululemon founder Chip Wilson in 2017.

Wilson started selling the pants for $100 at stores, which was a big risk. He did not know how receptive people would be, but he hyper-targeted the pants to well-educated, time-pressed women in their early thirties.

"Time is so critical to them that they don't have time to buy something and then return it," Wilson said about the customers he was chasing. "They don't have time to buy something, have it for three months, then go buy another pair."

Narrowing in on one segment of the market was the only way Wilson believed Lululemon could succeed. He called his target group "super girls." He didn't worry he was alienating other women in the process and would miss out on their business.

"If you start making something for everybody, then you make something for nobody in branding."

In fact, Wilson was narrowing in on 32-year-old women.

"We focused in on just that 32-year-old professional woman who owned a condo, and traveled, and was fashionable, and athletic," he said. "We didn't care for the 33-year old, or the 31-year old. We just built for that one person."

Courting controversy

Wilson took Lululemon public in 2005 but said he started to run into problems as the company grew in size.

"What I really lost was control of the company on a governance level once it went public," he said. Wilson said he wanted to move into new markets and make the brand "synonymous with mindfulness," but was stifled by the bureaucracy of the company.

"That mindset isn't necessarily aligned with creativity, or trying to describe a future that is not here yet," Wilson said.

He became a lighting rod for criticism along the way. He and his employees once dressed up as babies working with sewing machines to try to stay ahead of criticism that Lululemon relied on child labor.

"It was just a marketing insurance policy, let's call it," Wilson said. "Nobody could accuse us of child labor, because we agreed with it."

Wilson never recovered from his next blunder. After the company was forced to recall a portion of its trademark black pants in 2013, he told Bloomberg, "Some women's bodies don't work for the pants."

"It's really about the rubbing through the thighs, how much pressure is there over a period of time," he added.

The comments drew outrage. A video apology Wilson recorded didn't help either . He resigned as chairman of the board of directors shortly after.

Wilson said the comments continue to trail him, but he doesn't regret saying them.

"I don't think it would have been me if I would have answered any differently," he said.

Lululemon would be "worth 30, 40% more than it is now," Wilson said.

Moving mannequins

Wilson stepped down from the board in 2015, but said the company would be in a better position today if he were still around. He remains Lululemon's fourth-largest shareholder.

"I think that the company would be a mindfulness company mostly that would be deep into yoga. I think it would be far more global than it is now," he said. "The company would have been worth 30, 40% more than it is now."

Lululemon declined to comment on Wilson's criticism. The company has more than 140 stores outside the United States, and it has been a good year for Lululemon. The company's stock price has doubled in 2018.

Wilson continues to follow retail closely. He says he wants to be on the board of an athletic apparel company by the end of the year.

He believes the "athletic clothing market is now the defacto way people dress" and has staying power.

"I think the fashion people lost the market," he said. "We're not wearing baggy clothing. We're not wearing suits and ties. We're wearing clothing that's a little bit tighter."

And even though Wilson has no control over the company he founded, he still tries to leave his mark. He frequents Lululemons whenever he is in a new city and often times moves things around when no one is looking. "Like a mannequin for instance. I have always wanted a mannequin at a 45 degree angle so people could see the front and back," he said. "I have 1000 techniques on how to set up a retail store, and I love to do it."