Isaac and Stephanie Adams live in Richmond, Va., and are expecting a baby in June. Last year they decided to buy a house. With home prices and mortgage rates both at historic lows, it seemed the perfect time. Unfortunately, student loans stood in their way.

"We were looking at the market going, 'Oh my gosh, the market is awesome right now. We can get some great house that our payments will be, our loan will be great to set us up financially well for our growing family, and we just weren't able to do it, take advantage of that,'" Stephanie said.

Between the two of them, the Adams' student loan debt tops $100,000. They pay $1,100 a month for the loans, and that, coupled with the fact that Isaac was working a contract job, was enough to disqualify them from getting a mortgage.

Read More: How the Student Loan Crisis Drags Down Home Prices

Their story is getting ever more common. Total student loan balances nearly tripled between 2004 and 2012, according to a new survey from the Federal Reserve Bank of New York. Now $1 trillion in collective student loan debt is directly affecting the housing recovery.

Let our news meet your inbox. The news and stories that matters, delivered weekday mornings. This site is protected by recaptcha

"Short term, you see a decrease in the number of first-time home buyers," said Brian Coester of Coester Valuation Management. "You're going to see somebody who would have been able to afford a more expensive house maybe go for the lower version or the downgraded version."

First-time home buyers usually make up over 40 percent of the home buying population, but their share has hovered at or below 30 percent during this recovery, according to the National Association of Realtors. The student debt burden has forced many potential buyers to rent or to move back in with their parents.

"Without the student loan debt, a year and a half, two years earlier would have been the time I could have afforded to buy a house, and probably something a little bit bigger," Sophia Chaale said.

Chaale is facing $60,000 in student loans from graduate and undergraduate schools. She is paying $320 a month on a 30-year loan. Only after living at home for two years was she able to apply for a mortgage and put a down payment on a home. She is scheduled to close at the end of April.



"I consider myself lucky that I had a place where I could save, but what about other people who aren't originally from this area, who have to pay an extra $1500 a month in rent, and that rent money is not going to savings. How are they going to be able to save up or even to make that transition from renting to owning, in addition to all the student loan debt?" Chaale wondered.

The answer is that many won't. Adding to the burden is the fact that one-third of student loan borrowers are delinquent on their debts, according to the Federal Reserve report. That directly affects their credit rating and, in today's strict credit environment, will keep them out of the mortgage market for years to come.

"Long term it's going to really affect especially the upper end, because people aren't going to have the excess income to buy the jumbo property or buy that high end property," said Coester. "It' s going to affect home prices as a negative, as more of a cap, because it's really debt that they are servicing."

TheAdams had to delay their home purchase for a year, while they reorganized their student loan debt and while Isaac found permanent employment. They now have a contract on a house, but they feel like they got in just under the wire, as home prices are suddenly moving up rapidly.

"As long as this house closes, I don't think we missed out," said Isaac. "Rates are still fairly low, but I do believe as this year progresses, things will change."