A brutal fourth quarter for the stock market is exacting a substantial price on investor psychology, particularly for the mom and pop crowd.

Pessimism among retail investors is at the worst level in about 5½ years, according to the latest American Association of Individual Investors survey. The 48.9 percent of those who think the will be negative in six months is up 18.4 percentage points from last week's reading and is the highest since April 11, 2013.

The drop came just before the market turned a mild upswing in an otherwise nasty period.

Worries about rising interest rates, a testy trade showdown between the U.S. and China, and fears about slowing growth both domestically and abroad have hit Wall Street hard since early October. The S&P 500 is off more than 9 percent in the quarter, while the Dow Jones Industrial Average has fallen about 7.2 percent and the Nasdaq is off some 11.6 percent.