Other than watching reports of rising interest rates, mortgage lenders and brokers probably weren't very busy last week. Mortgage application volume fell 1.7 percent for the week, according to the Mortgage Bankers Association's seasonally adjusted report. Volume was 15 percent lower compared with a year ago.

Extremely weak demand for mortgage refinances has been driving the overall drain on mortgage lending. Refinance volume fell 3 percent last week and was 32 percent lower than a year ago. The refinance share of total applications dropped to 39 percent. To get an idea of just how weak that is, the refinance share of total mortgage applications was 62 percent two years ago.

Refinance volume is highly rate-sensitive, and interest rates are now about a full percentage point higher than they were one year ago. Last week, the average contract interest rate for 30-year fixed-rate mortgages with conforming loan balances ($453,100 or less) increased to its highest level since February 2011, 5.05 percent, from 4.96 percent, with points increasing to 0.51 from 0.49 (including the origination fee) for loans with a 20 percent down payment.

"Rates moved higher last week, driven by strong data on the job market, indicating that the Fed will continue to raise rates," said Joel Kan, an MBA economist.

Mortgage applications to purchase a home, while less rate-sensitive week to week, also fell, down 1 percent for the week but 2 percent higher than the same week one year ago. Since so much of the demand today is at the entry level, where buyers often have less wiggle room in their wallets, higher mortgage rates will cut into home sales. On a $300,000 home, today's rates will cost homebuyers about $200 more per month in payments.

In order to afford more home, more homebuyers are turning to riskier, adjustable-rate mortgages. They usually carry significantly lower interest rates for a short, fixed term but can then adjust higher.

"Since the end of August, the ARM share has increased to 7.3 percent from 6.1 percent, while the 30-year fixed rate has increased 25 basis points," Kan said.

One bright note for potential buyers who have been faced with higher prices and hotter bidding wars this year, higher interest rates could mean less competition from cash-heavy investors.

"It might cause investors to reconsider buying an investment property," Matthew Graham, chief operating officer of Mortgage News Daily, said Tuesday on CNBC's "Power Lunch." "Even those all-cash investors that would buy with cash and then refinance into an actual mortgage loan might not see as much cash flow from that, and that could open the door for first-time homebuyers to get a loan, assuming they can find a house."