NEW YORK (CNNMoney.com) -- The financial crisis has weighed heavily on American households, and millionaires are no exception, according to a report released Wednesday.

The number of American households with a net worth of $1 million or more, excluding the value of their primary residence, fell 27% to 6.7 million in 2008 from an all-time high of 9.2 million the year before, according to a report from market research firm Spectrem Group.

"America has a lot fewer millionaires than when this economic crisis began," said George Walper, president of Spectrem Group, in a written statement.

But don't weep only for the 2.5 million fewer millionaires. The report, which is based on surveys of 3,000 affluent households, also showed the number of both multi-millionaires and aspiring millionaires plummeted last year.

Affluent households, defined as those with a net worth of $500,000 or more, declined 28% to 11.3 million from 15.7 million.

Even the very rich have not been immune. Households worth $5 million or more, excluding primary residence, fell 28% to 840,000 last year from 1.16 million households in 2007.

"The culprit is not just the stock market, which we all know has dropped precipitously, but broad declines in the asset classes available to the nation's wealthiest investors," Walper said.

Respondents said the average value of investments in their principal residence, mutual funds, managed accounts and IRAs all fell in 2008 versus 2007, according to the report.

Millionaire households estimated that the financial crisis dented their net worth by between 30% and 40%. And the survey's measure of investor attitude fell to an all-time low.

With such a grim economic outlook, about 45% of respondents said they have made changes to their investment portfolios.

A majority of respondents said they are shifting capital into safer assets such as cash. But about 30% of those surveyed, mostly younger households, indicated that they are still buying stocks.

The report did not bode well for financial advisers. Only 36% of those surveyed said they have a satisfactory relationship with their primary financial adviser. That's down from 85% last year.