Large investment firms and individual investors made big profits over the past year betting markets would remain placid. Now they are suffering as stocks turn volatile.

Man Group PLC, the world’s largest publicly traded hedge fund, suffered a 4.7% loss in its $3.1 billion AHL Diversified fund on Monday. And New York-based R.G. Niederhoffer Capital Management Inc.’s $517 million Diversified fund, which trades short-term market moves and was up 16.2% this year at the end of January, saw those gains evaporate on Monday, an investor...