BOSTON (Reuters) - A Texas retirement system and Goldman Sachs Group Inc pulled a combined $584 million from Fisher Investments on Friday, after allegations that firm founder Ken Fisher made sexist comments at a conference.

Slideshow ( 4 images )

The withdrawals come despite an apology from Fisher and new diversity initiatives by his firm. Those efforts may help Fisher maintain its asset base, which stood at $114 billion this week and is concentrated among high net worth individuals.

But holding on to high-profile pension and fund contracts may now prove difficult. On Friday, a spokeswoman for the Texas Employees Retirement System said after a due diligence review sparked by Fisher’s comments it is withdrawing $350 million from Fisher, citing “our fiduciary duty.”

The money is being redistributed within a portfolio of stocks, said spokeswoman Mary Jane Wardlow.

In addition, a securities filing on Friday by Goldman Sachs states Fisher’s asset management unit will no longer be a subadviser to the $600 million Goldman Sachs Multi-Manager Global Equity Fund , effective on or around Oct. 31.

A Goldman Sachs spokesman did not comment on how much Fisher manages for the fund or give a reason for the change. Fisher was among 10 outside managers listed by the fund in a company disclosure as of March. Fisher managed $234 million for the fund, according to a source close to the matter.

Fisher did not comment on the moves, which bring to more than $3 billion the total withdrawn after comments Fisher made at an investor conference this month. Others that dropped his firm include Fidelity Investments and pension managers in Michigan and Iowa.

According to a conference attendee, Fisher made derogatory remarks about genitalia and picking up girls among other topics.

After a public backlash, Fisher apologized in a memo to his firm’s employees, and to the public.

“Some of the words and phrases I used during a recent conference to make certain points were clearly wrong and I shouldn’t have made them. I realize this kind of language has no place in our company or industry. I sincerely apologize,” Fisher said in a statement sent by a representative.