This year is shaping up to be a dismal one for so-called quant funds, typically some of Wall Street’s hottest investors.

At Two Sigma Investments LLC, the $45 billion firm’s flagship Compass fund is down 2.5% this year through May 31, fund investors say. In 2016, the fund climbed 10.33% for the year and 15% in 2015.

AHL Dimension, a $5.2...