Les Wexner, the billionaire retailer who made Victoria’s Secret one of the most famous brands on the planet, has sold control of the troubled lingerie company as it deals with declining sales and the fallout of a scandal involving Wexner’s ties to convicted sex offender Jeffrey Epstein.

Wexner, 82, announced on Thursday that he would step down as the chief executive and chairman of L Brands and the company would sell a majority stake of Victoria’s Secret to a private equity firm.

Wexner founded L Brands in 1963 with a $5,000 loan from an aunt. In an email to staff he referenced the company’s humble beginnings. “Today feels very similar to the day that my aunt offered me a new start,” he wrote. “Today is the beginning of an important new chapter in the evolution of the enterprise.”

Widely regarded as an extraordinarily talented retailer, the “Merlin of the mall” bought Victoria’s Secret in 1982 for $1m and built it into the largest lingerie retailer in the US. By the 1990s it had sales of more than $1bn and its famous – or infamous – annual fashion show, featuring models known as Angels, helped launch the careers of top models including Gisele Bündchen, Tyra Banks and Heidi Klum.

But sales faltered from 2015 onwards as more body-positive brands like Aerie and Lively and athleisure brands like Lululemon attracted shoppers tired of Victoria’s Secret’s padded, push-up aesthetic.

Les Wexner in Columbus, Ohio, on 19 September 2014. Photograph: Jay LaPrete/AP

Last year when the company canceled its annual show after years of declining viewership Stuart Burgdoerfer, the chief financial officer, said: “We think it’s important to evolve the marketing of Victoria’s Secret.”

The company’s internal culture has also been criticized. A New York Times investigation found an entrenched culture of “misogyny, bullying and harassment” at the company in which “abuse was just laughed off and accepted as normal”. L Brands did not dispute the report’s findings and said it has “made significant strides” in corporate governance and workplace practices.

Wexner’s reputation was also tainted by his long association with Epstein, the financier who died in a Manhattan jail cell in August while facing federal sex trafficking charges.

Epstein was a personal adviser to the retail chief, whose family has a personal fortune of $4.5bn according to Forbes. Epstein signed Wexner’s tax returns, was allowed to borrow money on his behalf and over the years obtained a portfolio of properties including a mansion in Manhattan and a luxury estate in Ohio worth roughly $100m that had previously been owned by Wexner or his companies.

Epstein was arrested for sex trafficking minors last year. L Brands then hired the law firm Davis Polk & Wardwell to conduct “a thorough review” of Wexner’s involvement with Epstein.

Wexner has said he is now “embarrassed” by his ties to the “depraved” Epstein and that he severed ties to him in 2007 after previous allegations against Epstein emerged.

As part of the deal Sycamore Partners, a firm that specializes in retailers, has bought a 55% stake in Victoria’s Secret for $525m. Wexner will become chairman emeritus of L Brands when the deal is finalized. L Brands’ bath shop chain Bath & Body Works will become a separate public company.