BERKELEY, Calif.—For Michael Kharitonov, building a hedge fund based on machine learning has been a rule of threes: It was three times as hard, and it took three times as long, as anticipated.

“Most of the things we’ve tried have failed,” said the co-founder of a little-known firm called Voleon Group.

Machine learning, a set of techniques...