The Columbus Day holiday may have closed some lending offices, but rising interest rates are more likely the culprit for weakness in the mortgage market.

Total mortgage application volume fell 7.1 percent for the week, according to the Mortgage Bankers Association's seasonally adjusted report. There was no adjustment made for the holiday. Volume was 15 percent lower compared with the same week one year ago.

Applications to refinance a home loan, which are highly sensitive to even the smallest rate moves, fell 9 percent for the week and were 33.5 percent lower than a year ago. Rates have moved 22 basis points higher in the past four weeks and have jumped 96 points in the past year. With fewer borrowers now able to benefit, refinance volume, which had been the majority of mortgage business following the recession, fell to 38.1 percent of total applications from 39 percent the previous week.

The average contract interest rate for 30-year fixed-rate mortgages with conforming loan balances ($453,100 or less) increased to its highest level since February 2011, 5.10 percent, from 5.05 percent, with points increasing to 0.55 from 0.51 (including the origination fee) for loans with 20 percent down payments.

"Treasury rates increased over the week, mainly as communication from Federal Reserve officials pointed to a continued path of rate hikes, based on the strength of the economy and hot job market," said Joel Kan, an MBA economist. "Furthermore, four out of the five rates tracked in our survey increased."

Mortgage applications to purchase a home also fell sharply, down 6 percent for the week. They were 2.5 percent higher compared with the same week one year ago. Homebuyer demand is strong, but affordability was weakening even before rates began to rise, as tight supply pushed home prices sharply higher. The combination now of high prices and rising rates is clearly throwing cold water on the heat in housing earlier this year.

Mortgage bankers meeting at the MBA's annual convention this week expressed concern about shrinking volume and shrinking profits, but the association's new CEO, Robert Broeksmit, had a brighter outlook.

"Well of course I'm concerned, but I'm optimistic because the economy is so strong and the millennials are out buying houses and the demand is so high. We do have to work on the supply side," said Broeksmit. "I think what you'll see is as the rates continue to tick up, home price appreciation, the pace will slow, and there will be an equilibrium over time."