Plenty of evidence, anecdotal and otherwise, shows that an increasing number of Americans expect to continue working past the age of 65. But that's not the ideal age, it seems, when people ponder their golden years.

Americans on average view 61 as the ideal age to leave the rat race behind, according to a survey released Wednesday by Bankrate.com.

Younger adults, particularly those 38 to 53, are more optimistic about their prospects of an early retirement, as Gen-Xers identified 60 as the prime age to stop working, while millennials, or those 18 to 37, view 61 as the right time.

Get Breaking News Delivered to Your Inbox

And in what's perhaps an indication of reality setting in, Americans 64 to 72 and those 72 and older are more conservative in their estimates of when to retire, picking 64 and 65 years old, respectively.

Broken out by gender, women estimate the ideal age to retire as 62, two years later than what men peg. Regionally, those living in the Northeast stood out from the rest of the country, listing 63 as the ideal age to retire.

A related data point: 59 percent of Gen-Xers believe one should ideally start saving for retirement by 21, which is 10 percentage points higher than everyone else, the poll of 1,001 in late June and early July found.

Having an aggressive retirement plan is great, so long as one starts the financial actions needed to bring it to fruition. "If you're striving to retire early, you need to start consistently setting aside money for the future right now," Bankrate analyst Amanda Dixon said in an emailed statement. "Time will be your greatest ally if you can get into the habit of saving money while you're young."

On the other hand, for those who don't start saving early, delaying retirement isn't a bad second option. After all, retiring at 70 instead of 62 means getting as much as 40 percent more each month in Social Security benefits.