The State of Michigan is breaking ties with long-time financial adviser Fisher Investments over crude remarks its founder and CEO made at a conference.

How much is it going to cost the investment firm? $600 million in pension funds it manages for Michigan.

CNBC and the Washington Post reported that the Michigan Department of Treasury's Bureau of Investments (BOI) terminated its contract with Fisher Investments because of comments that CEO Ken Fisher made at summit in San Francisco last Tuesday.

During a fireside chat at the conference, which is hosted by California-based Tiburon Strategic Advisors, Fisher compared his wealth management strategy to picking up women for sex, made explicit remarks about genitalia and made a reference to late financier Jeffrey Epstein, who was criminally charged with running a sex trafficking ring before committing suicide in August. The 68-year-old Fisher also spoke of doing acid and his belief that charities are immoral, the Washington Post reported.

Fisher has since apologized after his remarks became public.

In a letter obtained and posted by CNBC, Michigan Chief Investment Officer Jon Braeutigam addressed BOI board members Thursday, writing that Fisher's comments were "completely unacceptable." He said the BOI learned about the situation Wednesday from a news article and after having leadership discussions, the agency agreed that "prompt termination" was best.

"There is no excuse to not treat everyone with dignity and respect," Braeutigam wrote. "We have high expectations of our managers (and staff), not just with regards to returns but also in how they exhibit integrity and respect to all individuals."

The retirement funds will now be managed internally, Braeutigam wrote.

Fisher Investments was a domestic equity for the BOI for 15 years and was responsible for managing more than $600 million of the State of Michigan Retirement Systems funds. Braeutigam wrote that the investment firm's performance over the years was satisfactory and that he had never heard of Fisher making lewd comments before.

More:Young women feel left out of investment jobs. Here's what a group is doing about it

More:Detroit woman a leader in merger creating giant minority-owned investment bank in US

Since Fisher's speech at the Tiburon CEO Summit, there has been backlash within the business community, with three CEOs speaking out.

Alex Chalekian, founder of Pasadena, California-based Lake Avenue Financial, called Fisher's fireside chat "a true debacle" in a video on Twitter hours after the incident.

"Things said by Ken Fisher were absolutely horrifying," he said.

"Talking to some of the men and women, they were disgusted by this. Many of the women expressed to me this is one of the reasons why they don't like coming to these conferences. It makes them feel very uncomfortable and this obviously doesn't help the situation."

Sonya Dreizler, founder of impact investing company Solutions with Sonya, released a statement Wednesday on Twitter, confirming the things said in Chalekian's video were true.

"The conference content is supposed to be kept private, so that CEOs can be candid about internal business opportunities and challenges," she wrote. "Since this content is not about business issues, I'm choosing to break that code of privacy to confirm that the comments from the stage were indeed outrageous."

Rachel Robasciotti, founder and chief executive of wealth management firm Robasciotti and Philipson in San Francisco, also spoke out. Retweeting Chalekian's video Wednesday, she said, "I was in this room full of financial services CEOs when what Alex reports in this video was said — one of the most disgusting and disappointing moments in my career."

Tiburon addressed Fisher's comments Thursday, with Managing Partner Charles "Chip" Roame writing in an open letter that he was "extremely disappointed." In addition, he wrote that the sexist remarks continue to create less inclusion in the wealth and investment management industry, and that he had worked hard to find women and minority CEOs for the summit's 18-year history.

"These comments lacked the dignity and respect that should be expected by any Tiburon CEO Summit speaker or attendee," Roame wrote. "These were unacceptable words at Tiburon, in the wealth and investments industry, and in society generally."

Roame has since barred Fisher from speaking at future Tiburon conferences.

According to Forbes, Fisher founded his investment company in 1979 with only $250. Now, the businessman is worth $3.8 billion, has written 11 books and has a finance column with USA Today (the Free Press is part of the USA Today network).

“I have given a lot of talks, a lot of times, in a lot of places and said stuff like this and never gotten that type of response,” Fisher told Bloomberg on Wednesday, adding he thought his comments were taken out of context. “Mostly the audience understands what I am saying.”

He has since released a formal apology.

“Some of the words and phrases I used during a recent conference to make certain points were clearly wrong and I shouldn’t have made them,” he said. “I realize this kind of language has no place in our company or industry. I sincerely apologize.”