The Columbus Day holiday appears to have messed around with the mortgage numbers, but a week later the numbers are now clearer, and so is the message. Higher interest rates are hitting homeowners hoping to refinance and homebuyers hoping to get in on the fall housing market.

Total mortgage application volume rose 4.9 percent last week from the previous week, according to the Mortgage Bankers Association's seasonally adjusted index. This followed a large drop the week before, when the mortgage numbers were not adjusted for the Columbus Day holiday. Taking that out now, mortgage volume is lower than it was two weeks ago for refinances and purchases. Volume was also 16 percent lower than the same week one year ago.

Mortgage applications to refinance jumped 10 percent for the week but were 32 percent lower than a year ago, when mortgage interest rates were a full percentage point lower. Fewer and fewer borrowers are now able to benefit from a refinance because so many already locked in lower rates a few years ago. Those wishing to take cash out of their homes now are more likely to do a second home equity loan, rather than lose their low interest rate.

The average contract interest rate for 30-year fixed-rate mortgages with conforming loan balances ($453,100 or less) increased last week to its highest level since February 2011, 5.11 percent from 5.10 percent, with points decreasing to 0.52 from 0.55 (including the origination fee) for loans with 20 percent down payments.

Mortgage applications to purchase a home rose 2 percent for the week but were essentially flat compared to a year ago. Demand is strong, but affordability has weakened considerably. Home sales have been falling steadily all summer, even as more supply comes onto the market. Prices in the first half of the year were driven higher by a very short supply of homes for sale. The monthly payment on the average home is now 15 percent higher than it was a year ago, according to Zillow, due to higher home prices and rising rates.

"The holiday impacted refinance applications more than purchases," said the MBA's Joel Kan. "Meanwhile, purchase applications increased 2 percent over the prior week but were still 4 percent lower than two weeks ago — a sign that both the jump in mortgage rates and tight inventory continue to hold back application activity."

Mortgage rates were tested this week, as the stock market saw huge volatility, dropping over 400 points early Tuesday and then rebounding. While the yield on the 10-year Treasury, which mortgage rates loosely follow, did fall slightly, it was not enough for most lenders to reprice their rates.

WATCH: The hidden costs of buying a home