Can you spare $5 a day? If so, you could become a millionaire — one day.

"If you start in your 20s with a couple of reasonable investments, you can't avoid becoming a millionaire," said Michael Taylor, author of "The Financial Rules for New College Graduates."

However, many young people today put off investing.

More from Personal Finance:

What you need to know about the 'investor protection' rule

Tax holidays may not be the bargain you think they are

Not checking your credit report comes with risks

The average millennial doesn't expect to start saving for retirement until their late 30s, while half the generation isn't invested in the stock market at all, according to a study by TD Ameritrade.

Those findings make Taylor cringe.

He provided some math: If you save $5 a day in an account with a 10 percent annual return, you'll have around $30,000 in 10 years, $330,000 in 30 years and $2.3 million in 50 years. (The 's annual rate of return over the last 90 or so years has been around 10 percent. After adjusting for inflation, it's closer to 8 percent, however.)

Assuming a more modest 6.5 percent annual return, you'd have around $26,000 in 10 years, $168,000 in 30 years and $667,000 in 50 years.

To be sure, many young people are more worried about paying down their debt than building up their wealth.

The average millennial is $15,000 in debt, TD Ameritrade found. Student debt is increasingly a problem, with the average borrower paying almost $400 a month for their education.

But if you wait until you've paid off all your debt to invest, Taylor said, your goals will be harder to realize.

"If you start late, you will never catch up to the person who started early with the same amount," Taylor said.