The housing bust finally ended last year, with prices bottoming out and buyers starting to inch off the sidelines.

As in much of the economy, however, there’s a housing recovery for some but not for others. Wealthy buyers, you won’t be surprised to hear, are getting great deals on fancy homes — including vacation properties — which is fueling a boom in higher-priced homes. But at the lower end of the market, buyers remain scarce and sales are tumbling. “The housing market is still far from normal,” Bank of America Merrill Lynch economist Michelle Meyer wrote in a recent research note. “There is still a historically low share of first-time homebuyers.”

The housing recovery seemed to be enduring and widespread until last summer, when overall sales began to fall. That was right around the time that mortgage rates jumped. Combined with rising home prices, affordability suddenly declined. Sales of existing homes (which account for about 90% of all sales) have generally been falling ever since, as this chart shows:

View photos

The situation may be worse than the overall data suggest. It’s now clear that many home sales last year were to investor groups that bought thousands of foreclosures and other distressed properties in order to rent them out and profit from the monthly cash flow. Those sales have begun to drop off, as rising interest rates make such investments less profitable and the inventory of distressed properties thins out.

If more first-time or middle-income buyers were to materialize, that might make up for demand lost as investors retreat. But middle-class home buyers are precisely the ones who struggle to save enough for a down payment and get approved for a mortgage in a tight credit market. The result has been a bifurcation of the housing market, with sales of expensive homes going up while those of cheaper homes decline.

Data from the National Association of Realtors shows sales have fallen for homes priced below $250,000, but risen for those priced above. More specifically, sales of homes priced below $100,000 dropped 19% during the past year. For homes priced above $1 million, sales rose by 38%.

Vacation homes in high demand

Vacation homes are in high demand, with sales jumping 30% in 2013. That represented 13% of all transactions — the highest level since 2006, which was the last year of the housing bubble. And sales of new homes — which tend to be higher-priced properties targeted at wealthier buyers — have remained stable, even as existing-home sales have dropped. Houses are getting bigger, too: The average size of a new home is a near-record 2,607 square feet, even though fewer people live in the typical home these days.

Lavish homes aren’t the only high-end products selling briskly. Sales of BMWs are up 12% so far this year, after record sales in 2013. The glitzy auction house Christie’s had record sales in 2013. Upscale brands such as Nordstrom (JWN), Michael Kors (KORS), American Express (AXP) and Capital Grille are thriving, while struggles at Walmart (WMT), J.C. Penney (JCP), Sears (SHLD), Darden-owned (DRI) Red Lobster and many other mid-market brands reflect the financial stress felt by ordinary working Americans.

There’s no mystery behind the gilded spending of the wealthy. Financial portfolios crashed during the meltdown, with Americans’ total financial assets falling from a peak of $54 trillion in 2007 to $46 trillion in 2009. And nothing crimps the spending of the wealthy like a decline in wealth.

Story continues