If there's a case to be made for the Federal Reserve to hold fire on interest rate hikes, it's happening now in the housing market and the stocks that track the industry.

Homebuilder shares in the S&P 1500 recently sank 40 percent from their mid-January high. The move mirrors the early 2004 tumble that ultimately foreshadowed the coming housing collapse, which triggered the financial crisis, Bespoke Investment Group pointed out.

Source: Bespoke Investment Group

The most recent drop in the builders has come in tandem with the rise in the 10-year Treasury yield, the benchmark for most other rates. The note moved from a low of 2.82 percent on Aug. 22 to eclipse the critical 3 percent level, where it appears to be staying.

That, in turn, has coincided with a rise in mortgage rates to near 5 percent, which some experts say could be a major inflection point for the housing market.

"We know we're going to hold above 5 percent. The word is starting to get out," said Danielle DiMartino Booth, CEO of Quill Intelligence and an advisor to former Dallas Fed President Richard Fisher during the financial crisis. "That's one of those toxic things for residential real estate."

Real estate sales have been on a rough ride lately, with existing homes down 3.4 percent in September. Mortgage applications were down 7.1 percent for the most recent week.

With the talk of mortgage rates getting to 5 percent — Bankrate.com currently puts the average at 4.88 percent — has come chatter that the Fed will need to back off what looks like a pretty aggressive schedule for rate increases ahead.