Leslie Wexner, the billionaire behind Victoria’s Secret, said his former money manager Jeffrey Epstein misappropriated more than $46 million of his fortune, revealing for the first time some of the financial fallout from his relationship with the disgraced financier.

In a letter Wednesday, the founder and chief executive of L Brands Inc. wrote to members of his Wexner Foundation that the missing funds were uncovered after Mr. Wexner decided in 2007 to sever ties with Mr. Epstein. Mr. Wexner began the separation process after...