The furor in the financial world began almost immediately after Mr. Fisher spoke at an Oct. 8 conference sponsored by Tiburon Strategic Advisors, where, according to Bloomberg News, he said getting new clients was akin to “trying to get into a girl’s pants.”

Mr. Fisher’s initial response was to brush off the criticism. In an interview with Bloomberg the next day, he remarked that he had “said stuff like this” before and it had never caused a stir. But as the controversy grew, Mr. Fisher was forced to issue an apology, in which he said, “I realize this kind of language has no place in our company or industry.”

In a separate statement to clients and employees, Damian Ornani, Fisher Investments’ chief executive, said the firm valued diversity and was putting together a “diversity and inclusion task force.” The letter, obtained by The New York Times, did not address Mr. Fisher’s comments directly but referred to them as having “created an unnecessary distraction.”

Over the years, Mr. Fisher has not been shy about making statements that have drawn scorn.

At a Reuters investment summit in December 2011, Mr. Fisher said entrepreneurs had contributed more to the country than any politician and he went on to rank Lincoln as one of the worst presidents. Mr. Fisher said that Lincoln had chosen war over negotiations with the slaveholding states in the South.

Just last year, Mr. Fisher offered up similar statements about Lincoln and slavery on Twitter. Forbes uncovered posts on Twitter in which Mr. Fisher said that Lincoln was his least favorite president and that everyone in the country, including African-Americans, would have been better off if slavery had been allowed to end “peacefully.”

And CNBC obtained an audiotape of a 2018 conference in which Mr. Fisher compared selling a mutual fund to asking a woman in a bar for sex.

Mr. Fisher took an unconventional path to the upper echelons of the money management world. A native of San Francisco, he never worked at a traditional Wall Street firm and began his career in the private investment firm owned by his father, Philip A. Fisher, a well-known investor and author.