A quarter of borrowers who take out student loans end up defaulting within five years, while many of the people who pause or defer payments due to hardships end up paying more than they originally would've owed, according to a study from the Pew Charitable Trusts.

With total college debt at a record high $1.5 trillion, the findings highlight the challenges millions of Americans face in paying off their loans.

"We really need a repayment system that addresses this complexity and effectively leads to more positive outcomes," said Sarah Sattelmeyer, manager of Pew's project on student borrower success.

Get Breaking News Delivered to Your Inbox

The report focuses on student loan borrowers in Texas, but reflects what is happening more broadly across the U.S., Pew said. Of 400,000 residents in the state who took out a student loan between 2007 and 2011, roughly 24% defaulted within five years. Nationwide, 26% of borrowers defaulted, Pew said.

Student loan borrowers who default on their loans can face serious consequences, including collection fees, wage garnishment and damage to their credit scores. Payments from federal programs, like Social Security or tax refunds, can also be withheld. A borrower is technically in default after 270 days of missed payments.

Counterintuitively, perhaps, Pew found that students who borrowed the least — less than $10,000 — defaulted at higher rates. That suggests students who started college, but didn't complete their schooling, may have saddled themselves with debt, while struggling to land the higher-paying job that often comes with a degree.

Higher default rates also disproportionately affect students who attended for-profit colleges, as well as students of color and first-generation immigrant students.

Owing more than you borrowed

But even borrowers who don't default on their loans often end up pausing their payments, which leads to a higher debt load over time as interest charges continue to accrue on unpaid balances. The study found that 14% of student borrowers actually owed more than their original balance after five years.

Meanwhile, families on the hook for student loan debt feel the effects acutely in other ways.

"Many people who struggle to repay are feeling other types of financial distress," such as a job loss, Sattelmeyer said. "For many people who struggle with their loans, that's not the only element of their balance sheet that they're struggling with."

Sattelmeyer also said the findings underscore the need for the Department of Education to help people who fall behind on their college loans. Eliminating barriers to affordable payment plans would also help.