Fidelity has terminated a $500 million relationship with Fisher Investments, bringing the total yanked from the money manager to almost $1.8 billion.

Officials at the giant Boston-based asset manager confirmed it would end its relationship with Fisher on Monday, in light of inappropriate comments founder Ken Fisher made at an investment conference on Oct. 8.

Fisher managed $500 million in assets for Fidelity's Strategic Advisers Small-Mid Cap Fund.

"Fisher Investments does not provide investment advisory services for any portion of the assets of Strategic Advisers Small-Mid Cap Fund," said Vin Loporchio, a spokesman for Fidelity. "Assets previously managed by Fisher Investments have been reallocated within the fund."

While government-run pension funds make up a relatively small amount of the overall assets managed at Fisher Investments, $10.9 billion from 36 entities, according to the firm's regulatory filing with the Securities and Exchange Commission, how they respond may be a bellwether for other clients of the firm, industry experts say.

In all, Fisher had $94 billion in assets under management as of Dec. 31, 2018, according to their SEC filing. That figure reached $112 billion as of Sept. 30, 2019, according to the firm.

Fidelity is the fifth major institutional client to leave Fisher, based in Camas, Washington, since the billionaire made controversial remarks about sex at the Tiburon CEO Summit nearly two weeks ago. Previously, four government pensions pulled close to $1.3 billion from Fisher Investments.

CNBC obtained an audio recording of Fisher's comments at the Tiburon CEO Summit, as well as audio of him speaking at a previous conference. Clips from both were featured on CNBC's "Power Lunch." Combined, they show that the money manager made flippant remarks about sex.