The CEO of Victoria's Secret's lingerie division, Jan Singer, has resigned and plans to step down after working at the retailer for two years, a person briefed on the matter told CNBC.

The lingerie company, owned by L Brands, has struggled to grow sales as younger shoppers turn to more inclusive bra and underwear retailers. Singer joined Victoria's Secret from Spanx and had been trying to "aggressively" seek out new customers with new products and by understanding shoppers better.

L Brands shares initially fell Wednesday morning on the news of Singer's departure but were recently up about 1 percent.

The company is expected to share more details about Singer's departure on Monday, when L Brands reports quarterly earnings, said the person, who requested anonymity because the announcement isn't yet official. It wasn't immediately clear who Singer's replacement would be.

That person will be tasked with improving the brand's appearance to consumers. Female shoppers today are increasingly turned away by Victoria's Secret overtly sexy image, which is reinforced by its annual fashion show. The brand was made famous for its brightly colored push-up bras, but more women now are searching for comfort and fit. A handful of bra and underwear upstarts, like ThirdLove and Adore Me, are gaining ground by being more inclusive.

Shares of L Brands are down about 38 percent so far this year.

The Wall Street Journal first reported on Singer's departure Wednesday afternoon.

WATCH: Here's how Target is making a comeback