A contractor uses a hammer while working on townhouse under construction at the PulteGroup Metro housing development in Milpitas, California, Oct. 25, 2018.

Mortgage applications to purchase a newly built home dropped 11 percent in November, compared with a year ago, according to the Mortgage Bankers Association. Given the drop, MBA economists predict that sales of newly built homes in November fell 5 percent annually. Those numbers will be reported at the end of December by the U.S. Census.

This drop is surprising, given the sharp decline in mortgage rates in November, and therefore underscores the fundamental weakness in the housing market today. The average rate on the 30-year fixed mortgage started at 5.05 percent in the first week of November and fell throughout the month, ending at 4.86 percent by Nov. 30, according to Mortgage News Daily.

"Despite a still-strong job market and recent declines in mortgage rates, affordability challenges continue to hold back sales activity, as wage growth still lags behind home-price growth," said Joel Kan, associate vice president of economic and industry forecasting at the MBA. "Additionally, recent stock market volatility and some economic uncertainty likely also contributed to the pullback in home sales in November."

Stocks of the nation's homebuilders have been the positive outlier in a weaker market lately. That is because of the drop in mortgage interest rates. They are, however, still down sharply year to date.

The nation's largest homebuilders have been reporting weakness in buyer demand, and most are pointing squarely at affordability issues. While mortgage rates did drop back slightly in November, they are still higher than they were one year ago, and the expectation is that they will rise again going into 2019.

Homebuilder sentiment fell dramatically in November to the lowest level in more than two years, according to the National Association of Home Builders.

"For the past several years, shortages of labor and lots along with rising regulatory costs have led to a slow recovery in single-family construction," said the NAHB's chief economist, Robert Dietz, in a release. "While home price growth accommodated increasing construction costs during this period, rising mortgage interest rates in recent months coupled with the cumulative run-up in pricing has caused housing demand to stall."

Home price gains have cooled dramatically in the last six months, and there is growing concern among consumers that buying now would not be the best investment. In some markets, there is a fear that home prices will actually fall in the coming year, so buyers today would lose money.

"Lower house price expectations, tight inventory and a slowing economy will all act as a constraint on housing activity," according to a report from Capital Economics.

WATCH: How to use your home as a source of cash