A growing percentage of middle-class Americans say they have saved so little for retirement that they expect to work into their 80s or even until they either get too sick or die, according to a recent survey.

Nearly half of middle-class workers said they are not confident that they will be able to save enough to retire comfortably, according to a Wells Fargo survey of 1,000 workers between the ages of 25 and 75, with household incomes between $25,000 and $100,000.

As a result, 34% said they plan to work until they're at least 80 -- that's up from 25% in 2011 and 30% last year. An even larger percentage, 37%, said they'll never retire and plan to either work until they get too sick or die, the survey found.

Driving these concerns is that many of the respondents said they simply can't afford to pay their monthly bills and save for retirement at the same time.

Money 101: Planning for retirement

"For the past three years, the struggle to pay bills is a growing concern and the prospect of saving for retirement looks dim, particularly for those in their prime saving years," Laurie Nordquist, head of Wells Fargo Institutional Retirement and Trust, said in a statement.

The concerns come as many middle-class families are trying to make do with less. The country's median annual household income is down by more than 8% since 2007. And many of the jobs lost during the recent recession have been replaced with lower wage positions.

With minimal savings built up, a third of those surveyed said Social Security will be their primary source of income during retirement. Of those making less than $50,000, nearly half said they will rely mainly on Social Security.

In August, the average Social Security recipient received around $15,000 a year in retirement benefits, according to the Social Security Administration.

Related: Don't let the government's drama derail your 401(k)

Another factor holding back middle-class savers is a fear of investing in the stock market, said Nordquist.

Across workers of all ages, only 24% said they were confident in the stock market as a place to invest for retirement. And slightly more than half said they don't invest in the stocks because they are afraid to lose their savings in the ups and downs of the market.

This is despite the fact that financial planners say that investing in stocks is the best way to grow a nest egg that will be large enough to cover decades of retirement. Over years of savings, short-term losses are overtaken by the long-term gains that years of compounded returns offer.

"There is a striking amount of fear about the stock market among all investors," she said. "The middle class just isn't making the link between being invested and the potential growth of their savings.