Boomers and Gen X adults have swapped places on the retirement worry spectrum.

While boomers' anxiety has waned over the last few years, a new study by the Pew Research Center found more than half of Gen X are “not too” or “not at all” confident about retirement. Just a third of boomers echoed the sentiment.

It's a far cry from a similar survey three years ago, when only 19 percent of Gen X respondents said they weren't confident about retirement.

The reason is simple enough: Though much attention has been given to boomers whose nest eggs were pummeled by the recession, it was Gen X that saw their overall net worth fall the most.

"The median net worth of [Gen X] has fallen at a far greater rate than for any other age group both in the past 10 years and since the beginning of the Great Recession," the report says.

In fact, a census report released in June found Gen X saw a 59 percent decline in median household net worth between 2005 to 2010 –– the largest of any age group.



According to Pew, in inflation-adjusted dollars, they now earn $56,000 less than their counterparts in 2001.

On the other hand, 45- to 64-year-olds saw a $50,000 decline, and those over age 65 were the only age group whose net work grew over the last decade.

The study's findings are only more troubling when you consider how few Americans say they've managed to maintain their retirement savings over the last few years.

"If there are 5,000 people in this country [who] can afford to retire I would be astounded," Jane White, President of Retirement Solutions, LLC, told Business Insider.

"The actuarial rule of thumb is that you need to have accumulated a minimum of 10 times your pay at retirement. The typical wage for 65-year-olds is $65,000 and most have accumulated less than $100,000."

While ownership of stocks and retirement accounts, such as 401(k) and thrift accounts, was down among most age groups, according to Pew, ages 35 to 44 saw stock ownership fall by 9 percent.

Need tips on saving for retirement? See 4 ways to save even if you're flat broke >