The postelection euphoria for the economy seems to be cooling, and that is taking a toll on confidence in housing.

A monthly home purchase sentiment survey from Fannie Mae dropped in March, after hitting an all-time high in February. For most Americans, a home is their single largest investment, and both employment and income are major factors in deciding whether or not to buy.

In March, the share of Americans who reported that now is a good time to buy fell 10 percentage points, according to Fannie Mae. Consumers also reported dramatically less confidence in the stability of their jobs.

Those who reported that their household income is significantly higher than it was 12 months ago fell 8 percentage points compared with February. This as home price gains accelerate in most major markets, with some hitting new highs.

"Strong home price appreciation has turned into a double-edged sword for the housing market as it boosted the net share of consumers saying it's a good time to sell to a record high, surpassing the plunging good time to buy indicator for the first time in the history of the survey," said Doug Duncan, senior vice president and chief economist at Fannie Mae

The net share of Americans who say that mortgage rates will go down over the next 12 months fell 5 percentage points to a new survey low, even lower than it was during the so-called taper tantrum in 2013, when mortgage rates jumped decisively higher in just a few weeks. The fear of higher rates, however, could be positive.

"The market could get a boost from homebuyers who decide to jump into the market before rates rise further," said Duncan.

The housing market could still see a tail wind from more new listings this spring, but whatever the jump in supply, it is highly unlikely to meet current demand. Homebuilders are still operating below historical norms, partly because they can't find enough labor to put up the houses. The employment report released Friday showed only a very slight increase in construction jobs, not enough to make much of a difference.

"Although growth in wages and labor force participation would have been good news for the future of the housing market, the critical metric right now is residential construction jobs," said Nela Richardson, chief economist at Redfin. "This is the number to worry about from the homebuyer's perspective."