To be considered "rich," Americans say you need a net worth of at least $2.3 million.

That's according to a recent poll by SeniorLiving.org, which asked 1,000 U.S. adults how much a person would need to have in order to be called "rich." For those still working, you'd need to earn an annual salary of over $300,000 to earn the "rich" title.

In comparison, the median U.S. household income was $60,336 in 2017, according to the latest data available from the U.S. Census. The median net worth is $97,300, according to the Federal Reserve's most recent research, the 2016 survey of consumer finances. The average net worth is significantly higher: $692,100, according to the Federal Reserve.

The responses did vary slightly by generation. Baby boomers (ages 55 to 73) and Gen X (ages 38 to 54), unsurprisingly, named a higher net worth required to be rich, while millennials (ages 21 to 38) gave a slightly lower figure.

Millennials : It takes $2.2 million to be considered rich

: It takes $2.2 million to be considered rich Gen X : It takes $2.6 million to be considered rich

: It takes $2.6 million to be considered rich Boomers: It takes $2.6 million to be considered rich

The poll's results tracked with a similar survey, the Schwab Modern Wealth Index, which was released earlier this year. About 1,000 adults between the ages of 21 and 75 told Schwab that you needed a net worth of $2.27 million to be considered wealthy.

Regardless of what you think it takes to be rich, don't wait to start building up your wealth. You may not be rich today, but by saving up throughout your life, you can likely achieve real wealth and financial security.

"Building your net worth is a life's journey — it's not something that can happen overnight," says Farnoosh Torabi, personal finance author and host of the "So Money" podcast.

For some, there may be a limit to how much you can do with your money, especially if you have student loans, credit card debt and other monthly expenses. But "getting into the habit of saving" can be really simple and can produce long-term benefits, Torabi explains.

Make saving easy by setting up a regular, automatic transfer from your checking account to a savings account. This doesn't have to be a lot, maybe $5 a day or even $5 a week. "The key is to have it leave your hands before you get a chance to spend it," Torabi says. "Over the course of six months, a year or even just a little bit of saving at a time... [it] will accumulate and it will motivate you to do more."

Don't miss: Self-made millionaire: This is the No. 1 way to get rich—and most young people are not doing it

Like this story? Subscribe to CNBC Make It on YouTube!