Refinance volume led the way down, falling 4 percent from the previous week and 15 percent from a year ago, when rates were much lower. Borrowers looking to refinance are extremely rate-sensitive in today's market because so many people already refinanced to record low rates following the recession.

Total application volume fell 2.5 percent for the week and was 3.3 percent lower than a year ago, according to the Mortgage Bankers Association's seasonally adjusted report.

Both homeowners and homebuyers took a deep breath as rates continued to rise last week, pulling back a bit from the mortgage market.

The refinance share of mortgage activity decreased to 36.5 percent of total applications, its lowest level since September 2008.

The average contract interest rate for 30-year fixed-rate mortgages with conforming loan balances ($453,100 or less) increased to its highest level since September 2013, 4.80 percent, from 4.73 percent, with points increasing to 0.53 from 0.49 (including the origination fee) for 80 percent loan-to-value ratio loans.

"Market sentiments about strong domestic growth and higher inflation in the U.S. pushed the 10-year Treasury to the 3 percent mark last week, the first time since 2014 that yields have hit that level," said Joel Kan, an MBA economist.

Mortgage applications to purchase a home, although less rate-sensitive week to week, also fell 2 percent for the week but were 5 percent higher than a year ago. Buyers today are struggling with short supply and weakening affordability as home prices rise at a faster clip than any time in the past four years.

The adjustable-rate mortgage share of activity increased to 6.7 percent of total applications. ARMs offer lower rates, and demand usually rises as buyers try to afford more.

Mortgage rates have not moved at all this week, as financial markets were closed in other countries for the May Day holiday. That could change, with the monthly employment report at the end of this week and the Federal Reserve meeting Wednesday.

"While the Fed is not even remotely expected to hike rates again at this meeting, investors are always tuned in to the verbiage of the announcement in case it offers clues about the future policy path," said Matthew Graham, chief operating officer at Mortgage News Daily.