Some cracks may finally be starting to show in the foundations of what has been a very strong housing market.

Home Depot reported quarterly sales that missed forecasts Tuesday and the CEO specifically cited "a slow start to the spring selling season."

Shares of Home Depot (HD) fell about 2% in early trading on the news. Rival Lowe's (LOW), which will report its first quarter earnings on May 23, slipped 1% as well.

Home Depot's same-store sales, which measure how well locations open at least a year are doing, rose 4.2% in the quarter. But Wall Street was expecting a 5.4% jump. This was the first time Home Depot failed to top analysts' estimates in nearly two years.

Rising mortgage rates could be a problem for the company. The yield on the 10-year US Treasury bond, a key benchmark for mortgage rates, is back above 3% and is at its highest level since July 2011.

If mortgage rates continue to climb, that could make it more difficult for people to buy a house -- which would not be great news for Home Depot.

And Wall Street is starting to worry that the housing boom could be coming to an end too. Shares of big builders Lennar (LEN), Toll Brothers (TOL), KB Home (KBH) and D.R. Horton (DHI) all dipped Tuesday and they are each down between 10% and 20% so far this year.

Related: Here's what's killing Sears

The weather was also a factor in Home Depot's disappointing quarter.

While many retailers often trot out bad weather as an excuse for poor sales, it's a legitimate concern for a company that sells building products to people constructing homes outside.

The word "weather" was mentioned 14 times during Home Depot's conference call with analysts.

Home Depot cited "extreme winter weather" as something that hurt gardening supply sales. Sales in the northern part of the US and Canada were particularly weak.

The sluggish sales growth is particularly surprising since Home Depot has been one of the better-performing retailers for some time now.

Home Depot has outperformed the Dow and the SPDR S&P Retail ETF (XRT), by a wide margin for the past five years. Home Depot is a Dow component.

The company has benefited from the fact that many prospective home sellers have been looking to make improvements on their homes in order to help boost the sale price. Strong demand for new homes has helped fuel Home Depot sales too.

Related: Bed Bath & Beyond is in serious trouble

Home Depot also continues to be part of a small group of so-called "Amazon-proof" retailers.

Home Depot shoppers, particularly professional builders, are more likely to go to a store for plywood, lumber, concrete and other critical construction materials so they can actually see and touch them as opposed to simply buying them online from Amazon (AMZN).

And when customers are buying online, many prefer to go to the store to pick up the products. Home Depot's online sales rose 20% in the quarter.

So Home Depot's biggest challenge going forward isn't likely to be Amazon. It's rising mortgage rates.