During the housing boom of the 2000s, jumbo mortgages with very large balances became a flashpoint for a brewing crisis. Now, researchers are zeroing in on a related crack but in the student debt market: very large student loans with balances exceeding $50,000.

A study released Friday by the Brookings Institution finds that most borrowers who left school owing at least $50,000 in student loans in 2010 had failed to pay down any of their debt four years later. Instead, their balances had on average risen by 5% as interest accrued...