Virtually all investors have fled Portland, Oregon-based Common Sense Investment Management following the arrest of founder Jim Bisenius for soliciting a prostitute, sources close to the matter said.

The redemptions would leave the 22-year-old firm less than $150 million by year-end, assuming no investors change their mind, according to one of the people with direct knowledge of the situation.

Clients have requested to redeem more than 90 percent of the $3.2 billion the fund of hedge funds managed around the time of the arrest in late August , according to three people familiar with the situation.

Common Sense did not respond to requests for comment.

(Read more: Founder of $3 billion hedge fund busted in prostitution sting)

Among those who have redeemed recently include the Fresno County Employees Retirement Association, the Cincinnati Retirement System and the Oklahoma Municipal Employees Retirement Fund.

Pension consultant Arnerich Massena also recommended its clients leave. The firm declined to comment on who those clients were.

Not all the redemptions—Fresno County, for example—were directly because of the arrest.

"It's a tough issue. From our vantage it's a personal issue first and foremost and does affect the business," said Phillip Kapler, administrator of the $3.5 billion Fresno pension fund. "But if it affects the morale or key people at the organization and it becomes detrimental to our interest, it would be a problem."

Common Sense responded to the news a month ago by backing founder Bisenius, a 62-year-old active in Christian-related philanthropy.

(Read more: Once-$14 billionhedge fund confirms wind down)

"Jim Bisenius' recent personal transgression bears no reflection on this outstanding team of professionals or the quality of portfolio management at CSIM," firm president Dean Derrah said in statement on Sept. 4, the day the news broke nationally. The firm said Bisenius would remain chief executive officer and chief investment officer.

Most observers were not surprised at the investor reaction.

"The biggest reason people pulled wasn't to penalize them for the personal transgression. It was more that everyone else might get out," said a person familiar with the firm and its investors. "'Let's not be the last one to leave' was the mentality."

Others wondered if the business would survive.

"If the redemptions come as projected, this will call into question the ongoing viability of Common Sense," said another person who has tracked the situation closely, referring to the difficulty of running an asset management firm with such a small amount of client capital and fee revenue. "Even if you still believed in the investment strategy and had no problem with the extracurricular activities of the founder, you still need people there. I don't see how they come out of this."

Common Sense already had performance and personnel problems before the arrest.

The Common Sense Long Biased Offshore fund was up 8.3 percent this year through July 31, according to a report from The University of Toledo Foundation (which started redeeming in February). But the net annualized return for the fund since May 2008, when the foundation first invested, was just 1.8 percent, according to the report.

Long-biased funds focus on investments that will improve in value, rather than short-positions that gain on losses.

(Read more: Cohen's SAC,prosecutors find settlement elusive)

Two key employees—partners and portfolio managers Jonathan McGowan and Scott Kelly—also left in January this year to run 801 West Capital Management.

Common Sense assets had already declined from $3.9 billion in 2012 and $4.2 billion in 2011. The firm managed $3.2 billion in February.

"I'm not surprised their assets have dropped--there was already poor performance, lots of change in leadership," said one of the observers. "With this it was like 'OK, quit hitting your head against the wall.'"

—By CNBC's Lawrence Delevingne. Follow him on Twitter @ldelevingne.

