For more than a year, Jamie Chastain had $177 withheld from his Social Security check each month because of an unpaid student loan from three decades ago.

The 68-year-old has been struggling to pay off his debt since 1982 when he borrowed about $12,500 to get a Master's degree in Liberal Arts from the University of Southern Florida. It took him nearly a decade to find a steady job, and he defaulted on his loans before finding work at a community college as a media technician and humanities instructor. The government garnished his wages by about $400 a month throughout most of his career to collect the debt.

Chastain estimates he's paid back three times what he originally borrowed. But because of the interest, the loans still weren't paid off when he retired in 2014. So the government began garnishing his Social Security checks.

There are more people over the age of 65 with student loan debt than ever before, according to a new report from the Government Accountability Office.

And those who are in default may have faced decades of reduced wages and benefits. In 2015, the government reduced Social Security checks for a total of 173,000 Americans, up 380% from 36,000 in 2002. That includes younger people receiving Social Security disability payments as well.

Related: Should I save for retirement or pay off student loans?

The growing number of seniors with student debt has gotten attention from some lawmakers.

"This report shows us that seniors clearly aren't immune to the student loan crisis -- they're deeply impacted by this issue to the point that it's leaving many of them in a dire financial situation," said Senator Claire McCaskill.

The GAO recommends changing the rules so that Social Security checks can't be reduced as severely. Over the past decade, the number of seniors whose Social Security benefits have been garnished to below the poverty line because of unpaid student loans has increased from 8,300 to 67,300, the report said.

That's because there are more people in default, but also because the threshold for Social Security garnishments set in 1998 was never adjusted for inflation. The GAO recommends raising that threshold, which was set to prevent undue financial hardship for borrowers who "may be unable, rather than unwilling, to repay their debts." This would require Congressional action.

Related: This is how much student debt will cost you in retirement

The report also suggests that the garnishments are not successfully collecting outstanding debt. More than one-third of older Americans were still in default after two years of reduced Social Security payments.

Part of the problem is that only a small portion of garnished payments went toward paying off the loan principals. Most of the money covered interest and the $15 monthly fee the government charges to reduce Social Security benefits.

Withholding part of a borrower's federal tax return -- another way the government collects debt in default -- was more effective and charged a lower fee.

"The hard-earned Social Security checks that are the sole source of income for millions of seniors should not be siphoned off to pay interest and fees on student loan debt," said Senator Elizabeth Warren.

Related: Our sons' student debt is delaying our retirement

The GAO also recommends the government simplify the disability discharge application process. Those who can no longer work because of a disability are eligible to get their outstanding debt wiped away, but the report found that the process is difficult.

This kind of change could be made by executive action and would not require Congress to act. The Obama Administration has made efforts to make the discharge process easier. Earlier this year, it started to proactively notify those who are disabled and might be eligible for the program.

Chastain was one of the 387,000 borrowers notified because of a recent diagnosis that affects his ability to work. He applied for relief and was approved for a discharge this summer. The garnishment of his Social Security checks stopped and the outstanding $12,000 in student debt was wiped away.

He says he wasn't looking for a free pass, but the system seemed "rigged" against him.

He's happy to have the money back in his pocket each month, but is apprehensive about what comes next. The discharge will be taxed as income, so he's saving the extra money until he knows how much he'll owe the IRS and when. It could be more than $2,000.

"I'm just one little guy and there are plenty of people worse off than me that have gotten hammered," Chastain said.