Two Sigma Investments LP spent the past 15 years honing its quantitative approach to investing. Now, it is turning its number-crunching prowess on new prey: insurance.

The firm is joining with American International Group Inc. and Hamilton Insurance Group Ltd. to provide an automated, online analytical system to issue policies in minutes to smaller businesses like clothing shops, beauty parlors and medical offices.

Under...