Kenneth Fisher, chief executive officer of Fisher Investments, speaks at the Forbes Global CEO Conference in Sydney, Australia, on Tuesday, Sept. 28, 2010. Gillianne Tedder | Bloomberg | Getty Images

The City of Boston is ending its relationship with Fisher Investments, pulling $248 million in pension assets from the firm. Mayor Martin Walsh announced on Wednesday that the city would stop working with the company in light of sexist comments Ken Fisher had made at an investment conference last week. "The statements made by Ken Fisher implicate not only his own judgment, but potentially that of the company as a whole," Walsh wrote in a letter to the Boston Retirement Board. The board today voted 5-0 to end its relationship with Fisher Investments "While there are no doubt employees of the firm that are just as disturbed by these comments as I, there remains a risk that such thinking runs deeper than this specific commentary, and this is not a risk to which I believe the Retirement System should expose itself," wrote Walsh.

The Boston Retirement System has a $5 billion portfolio in total. The pension plan's board will determine how to reinvest the $248 million that were divested, said Emme Handy, chief finance officer for the City of Boston. It's not uncommon for pension managers to spread investments across multiple firms. Boston is the latest pension plan to pull its assets from Camas, Washington-based Fisher. The state of Michigan is withdrawing $600 million of its pension fund from the firm, as well as Philadelphia's board of pensions, which yanked $54 million. Fidelity Investments said on Tuesday that it was reviewing a $500 million relationship with the firm. "We are very concerned about the highly inappropriate comments by Kenneth Fisher," said Fidelity spokesman Vincent Loporchio. "The views he expressed do not align in any way with our company's values." Ken Fisher has since apologized for the comments. "Some of the words and phrases I used during a recent conference to make certain points were clearly wrong and I shouldn't have made them," he said in a statement. "I realize this kind of language has no place in our company or industry. I sincerely apologize."

Tiburon comments

CNBC obtained an audio recording last week of Fisher's comments at the Tiburon CEO Summit, as well as audio of him speaking at a previous conference. Clips from both were featured on CNBC Power Lunch last Friday. Combined, they show that the money manager made flippant remarks about sex. In the audio obtained by CNBC, Fisher says at the Tiburon conference: "Money, sex, those are the two most private things for most people," so when trying to win new clients you need to be careful. He says: "It's like going up to a girl in a bar … [inaudible] …going up to a woman in a bar and saying, hey, I want to talk about what's in your pants."