When you need cash, there’s nothing like the temptation sitting in your retirement account — that stash of money that becomes all-too-available when you lose or quit your job. But withdrawing those savings can leave a deep scar on your future retirement income, according to a new report.

A hefty 35% of people who left their jobs last year cashed out their retirement savings rather than keeping their money in a tax-qualified retirement account, according to a new study by Fidelity Investments of the 12.5 million participants in the 401(k) plans it administers — and those findings are echoed by previous studies from other sources. (Note that just 806,000, or 6.5%, of that total number of plan participants ended their job in the January through September period studied by Fidelity.)

There’s no doubt that taking the cash is a financial necessity for some people, and it’s not surprising that the practice is more prevalent among those with lower incomes than among those who make more: 50% of those earning $20,000 to $30,000 cashed out in 2013, versus 13% of those earning $100,000 or more, according to Fidelity.

Younger savers are also more prone to the practice: 44% of those aged 20 to 29 cashed out, compared with 26% of those 50 to 59, the report said.

But anyone who doesn’t absolutely need the cash should think long and hard before grabbing that money to spend on stuff now. Generally, you’ll play income taxes and a 10% penalty for pulling out the money early. Plus, your long-term savings will take a hit. Read: 8 secrets of success from early retirees.

A hit to retirement income

Fidelity offered this hypothetical example: A 30-year-old who cashes out an account worth $16,000 would owe about $3,200 in taxes plus $1,600 in penalties — meaning he takes home only $11,200.

Investing with your conscience

Even more worrisome: If he takes that money out of his retirement savings, his future retirement income would drop an estimated $470 a month (in today’s dollars, assuming a real return of 4.7% and assuming he retires at 67 and dies at 93), according to Fidelity’s analysis.

A separate study found that people who cash out their plan could see their retirement income reduced by anywhere from 11% to 67%. That’s according to a projection by the Employee Benefit Research Institute, for a 2011 study of more than 1.8 million workers by consulting firm Aon Hewitt. (The projection varies based on how many different jobs the worker ends up in, and whether the new plans use automatic enrollment, among other factors.) Read the report here.

The Aon Hewitt report found that a whopping 42% of workers who left their jobs in 2010 cashed out of their retirement plan, with that percentage rising to 53% among workers age 20 to 29, and dropping to 32% for workers aged 50 to 64.

A Vanguard report in June 2013 found that 31% of 401(k) plan participants who left their job in 2012 cashed out at least part of their savings. That was up from 29% in 2007. Read the Vanguard report here.

“Given the ongoing uncertainties in the employment market, we anticipate an increase in cash-outs in the near term, under the assumption that plan savings are sometimes used as a personal form of unemployment insurance,” the Vanguard report said.

Don’t rush

If nothing else, if you’re leaving a job, at least take some time before making a decision relative to your retirement funds. Generally, you can let your 401(k) savings sit in your former employer’s plan.

“They probably shouldn’t look at touching their 401(k) from Day One. Some people go into that quickly. They may not understand all of the implications,” said Beth McHugh, vice president of market insights at Fidelity Investments.

“Keep it in the plan for now, until you can make a more thoughtful decision,” she said. “When you’re changing and advancing your career, you want to make sure you’re advancing your retirement as well, and that you’re not taking a setback every time you move to a new employer by cashing out.”

Your alternatives to cashing out may include keeping it in your prior employer’s retirement plan, moving it to a new employer’s plan or rolling it into an IRA.

Consider your options carefully: Research suggests that retirement savers are often urged to roll their money into an IRA — even when sticking with an employer plan might be better, and cheaper, for them. Read: Beware these IRA rollover mistakes.

Good news: Average balance almost doubles

The Fidelity study also found that the average 401(k) balance in 2013 jumped 15.5% over the year to $89,300 — almost double the average balance in March 2009 — $46,200 — when the stock market hit a low before starting to climb again.

The stock market’s rise in 2013 contributed 78% of that annual gain in the average balance, while 22% of the increase was due to employee and employer contributions.

The average account balance, still, is small, but it includes new and younger savers. For workers aged 55 and older, the average balance at the end of 2013 was $165,200, up from $88,700 at the end of March 2009, according to Fidelity.

The picture improves even more for those who have both a 401(k) and an IRA: Their average balance is $261,400, up 16% from $225,600 a year ago. This data is based on people who have a 401(k) and an IRA (of any type) managed by Fidelity.

And the picture brightens considerably for older savers with a 401(k) and IRA at Fidelity: The average balance for people 55 and up is about $380,000, while for those aged 65 to 69 it’s almost $447,000.

That’s getting a little closer to the $600,000 Mr. Money Mustache retired on in his 30s. Read more: How to retire 35 years early.

Related stories:

5 best money strategies for boomers

Your heirs want this even more than your money

You’re invited to ... Bitcoin: Boom and Bust