The hedge-fund business -- among the most reliable fortune-producing machines in recent years -- is going through a brutal shakeout.

Just a few years ago, traders found it relatively easy to quit Wall Street jobs, hang out a hedge-fund shingle and cash in. Investors beat down the doors with eagerness reminiscent of the late-1990s dot-com...