Investors borrowing record sums to bet on stocks exacerbated this month’s selloff, after they were hit with calls to reduce those obligations and forced to sell shares to raise cash.

If that debt, known as margin loans, continues to rise at the current pace, analysts warn that big selloffs and sudden bouts of volatility in the stock market could become more commonplace.

Retail...