That rate hit 3.34% for one day in the summer of 2016, before spiking much higher that fall. Before that, rates were this low in 2012. While rates generally follow the 10-year yield, there are certain market factors that keep rates above a certain level.

"Aggressive lenders will be at 3.25% today, and 3.375% will be the new going rate for the average lender," said Matthew Graham, chief operating officer at Mortgage News Daily.

The average rate on the popular 30-year fixed mortgage hit 3.34% on Monday, according to Mortgage News Daily. That is for borrowers with strong financials and credit scores.

As coronavirus fears hit financial markets, U.S. bond yields are tanking, pushing mortgage rates that loosely follow the 10-year Treasury yield toward an eight-year low. They could sink even lower.

"When rates fall this quickly, it's not so much that big banks draw the line on mortgage rates, but rather, the underlying Mortgage Backed Securities (MBS) market refuses to improve as quickly as the Treasury market," said Graham. "Both mortgages and Treasuries are feeling the impact of coronavirus panic. That's pushing rates lower. But mortgages also become less valuable to investors if they get paid off too quickly."

And those payoffs, or refinances, are surging right now. Applications to refinance a home loan are up around 165% annually, according to the Mortgage Bankers Association.

Mortgage applications to purchase a home have not been as strong, due to the severe shortage of homes for sale. Builders, however, may be getting a boost, especially those putting up more affordable homes.

Another barrier to entry for some buyers is still-tight lending standards. Sean Dobson, CEO of Amherst Holdings, which does have a mortgage arm, said tight lending is why his company got into the single-family rental business.

"Unless you have a large down payment or unless you have a very solid amount of free cash flow that's underwritable, and we forget about this because the Uber driver might not have income that is fungible from a mortgage lenders perspective, or the people working 3 or 4 jobs, or the contributors to CNBC who contribute to a few places, they may literally have trouble qualifying for a mortgage," said Dobson.