America’s pension crisis is growing — and a lot of people are pointing fingers.

Public pension funds don't have nearly enough money to pay for the benefits promised to government workers.

The shortfall across the United States grew by $295 billion between 2015 and 2016, according to a recent report from the Pew Charitable Trusts. And a handful of cities and states, including California, Illinois and Kentucky, must cover enormous costs if they are going to make good on the promises made to public employees as they retire.

Related: States have a $1.4 trillion pension problem

The soaring costs have fueled outrage in some camps, sparking a belief shared regularly in letters to the editor and conservative magazines that pensioners are living richly on the backs of taxpayers who themselves can barely afford a secure retirement, if any.

But there are a few misconceptions floating around this debate.

First, most retired public servants aren’t living that comfortably on their pension income alone. Sure, there are a few eye-popping payouts, like the $76,111 monthly check collected by an eye surgeon in Oregon, that the New York Times recently wrote about. But the National Institute on Retirement Security analyzed Census data and found that the average state and local pension benefit was $27,415 a year in 2016. Annual payouts ranged widely by state, from $16,441 in West Virginia to $37,934 in Connecticut.

Often, that’s the only income they have coming in during retirement — most state and local employees are not eligible for Social Security.

Second, the majority of that income isn't supplied by taxpayers at all. Between 1993 and 2014, about 64% of pension funding came from investment earnings, according to Census Bureau data.

As Pew noted in its report, state and local governments made overly rosy assumptions about the market returns their funds would make. The median public pension plan’s investments returned about 1% in 2016, well below the median assumption of 7.5%. That disparity added about $146 billion to the debt, the report found.

State and local contributions were also comparatively modest. On average, the National Association of State Retirement Administrators calculates that pension contributions account for 4.7% of state and local spending. That’s up from a low of 2.3% in 2002, when many governments weren’t contributing enough to meet their future obligations.

And third, not all states are struggling to meet their pension obligations. According to Pew, the market value of the assets in the state public employee retirement funds was 66% of the amount needed to pay promised benefits nationwide. But a handful of states, including New York, Wisconsin, Tennessee and South Dakota, had enough assets to meet more than 90% of liabilities.

Related: States are changing teacher pension plans

“States that have funding problems most often can blame it on lawmakers failing to make the required contributions and then they fall way behind,” says Kelly Kenneally, spokeswoman for the National Institute on Retirement Security.

Pension costs are indeed a challenge for many local governments. But it’s important to recognize how they arrived in that position and the costs to future retirees when changes are made.

And in recent years, changes have been common. According to NASRA, “nearly every state” revised their benefits policy in an effort to control costs after the Great Recession. Now, calls are mounting to switch public employees into cash balance plans, a hybrid between a pension and a 401(k). The plans are generally less generous because they calculate benefits based on the beneficiary’s lifetime earnings rather than their salary right before retirement, when their earnings are typically at their highest.

Graphic by Tal Yellin/CNN