An Automobili Lamborghini SpA Aventador vehicle sits on display during the Gallery event at the MGM Grand Detroit ahead of the 2019 North American International Auto Show in Detroit, Jan. 12, 2019.

The rich have cut their spending on everything from homes to jewelry, sparking fears of a trickle-down recession that starts at the top.

From real estate and retail stores to classic cars and art, the weakest segment of the American economy right now is the very top. While the middle class and broader consumer sections continue to spend, economists say the sudden pullback among the wealthy could cascade down to the rest of the economy and create a further drag on growth.

Luxury real estate is having its worst year since the financial crisis, with pricey markets like Manhattan seeing six straight quarters of sales declines. According to Redfin, sales of homes priced at $1.5 million or more fell 5% in the U.S. in the second quarter. Unsold mansions and penthouses are piling up across the country, especially in ritzy resort towns, with a nearly three-year supply of luxury listings in Aspen, Colorado, and the Hamptons in New York.

Retailers to the 1% are faring the worst, with famed Barney's filing for bankruptcy and Nordstrom posting three consecutive quarterly declines in revenue. Meanwhile, Wal-Mart and Target, which cater to the everyday consumer, are reporting stronger-than-expected traffic and growth.

At this month's massive Pebble Beach car auctions, known for smashing price records, the most expensive cars faltered on the auction block. Less than half of the cars offered for $1 million or more were able to sell. But cars priced at under $75,000 sold quickly — many for far more than their estimates.

In the first half of 2019, art auction sales were down for the first time in years. Sales at Sotheby's dropped 10% and Christie's auction sales were down 22% from a year ago.

There are many reasons for spending declines — tax changes, for instance, are to blame for some of the real estate slump.

And parts of the high-end economy have retained their shine — from luxury new car sales to Swiss watches and fashion.

Yet recent data suggest that the U.S. wealthy are beginning to shut their wallets. If their spending falls further, the broader economy could start to feel the pain. The top 10% of earners account for nearly half of all consumer outlays, according to Mark Zandi, chief economist at Moody's Analytics. But their spending has fallen over the past two years, while spending for the middle class has accelerated.

"If high-income consumers pull back any further on their spending, it will be a significant threat to the economic expansion," Zandi said.