While more people think now is a good time to buy a home, according to a monthly sentiment survey from Fannie Mae, more people also think mortgage rates will go up, and more people are concerned about keeping their jobs and growing their incomes.

"Five percent is definitely an emotional level inasmuch as it scares prospective buyers about how high rates may continue to go," said Matthew Graham, chief operating officer of MND.

The average rate on the 30-year fixed loan sat just below 4 percent a year ago, after dropping below 3.5 percent in 2016. It just crossed the 5 percent mark, according to Mortgage News Daily. That is the first time in eight years, and it is poised to move higher. Five percent may still be historically cheap, but higher rates, combined with other challenges facing today's housing market could cause potential buyers to pull back.

Millennials are in their prime homebuying years, and they're used to cheap credit. So they might be in for a rude awakening as mortgage rates jump.

Home sales have been sliding for much of this year, and total annual sales are expected to come in lower than last year. Affordability is the clear culprit. With rates now more than a full percentage point higher than a year ago, that adds at least $200 more to a monthly mortgage payment for a $300,000 loan. It also knocks some borrowers out of qualification because lenders are strict on how much debt a borrower can carry in relation to his or her income.

"For buyers that are inclined to buy now, they're doing the same things they've been doing: Looking at the appalling lack of available inventory, seeing if any of it meets their needs, and deciding whether or not they can afford the monthly payment. In that sense, 5 percent takes some buyers out of some markets, and it plays a supporting role in the leveling-off process that's already well underway for home sales," Graham said.

The stronger economy has definitely fueled demand for housing, but the last few years of near record-low supply pushed prices too high too fast. So while more consumers may want to become homeowners, more and more simply can't afford it.

"The economy seems to be coasting upward. But this kind of complacency, this kind of confidence can be volatile," Robert Shiller, co-creator of the S&P CoreLogic Case-Shiller Home Price Indices, said in an interview on CNBC's "Squawk on the Street." "The question is, is this a turning point in the housing market. I'm a little bit worried about that but not ready to call that."

Higher rates, however, could throw cold water on today's overheated home prices, as sellers see demand fall off and their houses sit on the market longer. The number of price cuts is rising quickly, signaling that sellers, and their expectations, are in fact coming back down to earth. Prices are unlikely to fall, but the annual gains should shrink.

Higher rates may also take some competition out of the buying market, which has been replete with investors looking to either flip or rent homes. A growing share of those investors use mortgages.

"One of the best unforeseen benefits of higher rates is that it makes for less enticing cash flow for investors. That means first-time homebuyers who can actually find a house they can afford are now more likely to have an offer accepted whereas they ran a much bigger risk of losing out to an investor in the past five years," said Graham. "Even all-cash investors typically refinance shortly after buying the investment property, so they can put that cash to work buying additional properties."

WATCH: Mortgage rates at highest level in 7 years