BOSTON (Reuters) - Fidelity Investments on Tuesday said it was reviewing its relationship with money manager Ken Fisher while Philadelphia’s board of pensions terminated its business with his firm after he allegedly made disparaging remarks about women last week.

FILE PHOTO: A sign marks a Fidelity Investments office in Boston, Massachusetts, U.S. September 21, 2016. REUTERS/Brian Snyder/File Photo

Known to investors through frequent television ads for his investment advice, Fisher had already lost business with Michigan’s retirement system last week and faces scrutiny from other similar funds, including in Florida, even as he and his firm try to make amends.

Fisher’s setbacks come in the wake of the #MeToo movement, which has exposed widespread sexual harassment or abuse of women in multiple spheres of American life and ended the careers of dozens of powerful men in media, politics, entertainment and business.

Philadelphia’s board of pensions terminated Fisher Investments as an investment manager because of “inappropriate statements” made at a conference last week, a city spokesman said on Tuesday. The city’s retirement system had roughly $54 million under management with Fisher, the spokesman said.

Fidelity Investments criticized the money manager over the remarks, saying it was reviewing the $500 million Fisher’s firm manages for the mutual fund giant.

“We are very concerned about the highly inappropriate comments by Kenneth Fisher. The views he expressed do not align in any way with our company’s values,” Fidelity spokesman Vincent Loporchio said. “We do not tolerate these types of comments at our company.”

In addition a spokesman for the Florida State Board of Administration, for which Fisher manages $175 million in assets, said the retirement system is reviewing his role.

In a memo to his firm’s employees sent to Reuters by a spokesman, Fisher said “It pains me to know that my comments have caused you grief, concern, and indignation. I sincerely apologize.”

In a separate employee memo, Fisher Investments Chief Executive Damian Ornani wrote that “Ken’s comments were wrong,” and said the firm is launching a task force to address diversity and inclusion at the firm itself.

Fisher Investments manages money within the $8 billion Fidelity Strategic Advisers Small-Mid Cap Fund. His firm manages more than $100 billion, including at top U.S. public pension funds.

Fidelity Chairman Abigail Johnson, who is one of the few female leaders among large financial service firms, told staff two years ago the company had no tolerance for any type of harassment after the dismissal of a well-known money manager who had been accused of misconduct.

(For a graphic, click here)

Alex Chalekian, chief executive of a financial advisory firm, called attention to the comments Fisher made at a financial conference last week in a video posted Oct. 9 on Twitter.

He said Fisher had made derogatory comments about genitalia, picking up girls and financier Jeffrey Epstein, among other topics. Epstein committed suicide in August while being held in jail awaiting trial on sex trafficking charges.

Charles Roame, managing partner of conference organizer Tiburon Strategic Advisors, said in a note on the firm’s website that comments made at the Oct. 8 conference “lacked the dignity & respect that should be expected by any Tiburon CEO summit speaker or attendee.”

Although he did not name Fisher, Roame praised Chalekian for going public “with his views of offensive behavior.”

Michigan’s $70 billion retirement system dropped Fisher Investments over the remarks on Thursday, according to a letter sent to the state’s investment board. Fisher had managed $600 million for the retirement system, money that will now be managed internally.

Fisher Investments is also a sub-adviser on the $600 million Goldman Sachs Multi-Manager Global Equity Fund. Goldman Sachs spokesman Patrick Scanlan declined to comment.

Teir Jenkins, investment officer for the city of Oakland, California, said retirement officials there had decided to replace Fisher Investments as manager of about $17 million in July. He said Fisher’s comments last week may move up the schedule.

“This kind of lights a fire under us. We are working to expedite the transition,” Jenkins said.