The stock market is banging out record highs this summer. So why isn't America celebrating?

The Dow Jones industrial average climbed above 17,000 for the first time Thursday, boosted by strong job gains. It's been a remarkable summer for stocks, but many people don't feel like they're seeing any benefit.

"Who cares what records Dow Jones hits, it has zero reflection of our horrible economy," wrote one Twitter user this week.

Get Breaking News Delivered to Your Inbox

There's a reason that people feel so disconnected from Wall Street. Only a fraction of Americans directly own stocks, and they tend to be wealthier. Most people have drifted away from the market over the years, burned by the financial crisis, economic recession and a feeling that the investing deck is stacked against them.

The upshot? Although financial markets are soaring, relatively few people are reaping the gains.

Just 15 percent of families directly owned stocks in 2010, according to a 2012 report from The Federal Reserve. That's down from a peak of 21.3 percent in 2001. Although millions of Americans are exposed to the market through retirement accounts and other assets, millions aren't -- as of 2010, only 51 percent of private-sector workers in the U.S. had access to a retirement plan at work, according to public-policy think-tank Demos.

Meanwhile, the number of people directly trading in stocks -- and who stand to gain or lose the most in market swings -- is dwindling.

In fact, the majority of Americans have no money in the stock market at all, including retirement accounts. Some 53 percent of Americans avoid the market completely, according to a Pew Research survey last year.

The richer and more educated people are, the more likely they are to invest in the stock market. Some 80 percent of households earning $75,000 a year or more are in stocks, including retirement accounts, Pew found, while just 15 percent of those earning less than $30,000 have invested.

The people who stayed in stocks have watched them surge recently, more than making up for the losses seen after the financial crisis that began in 2007. The stock market gains over the last few years have helped the wealthiest recover from the economic recessions. The richest 5 percent had 24 times as much money as the median household last year, The Associated Press reports. That's up from 16.5 times as much in 2007.

The gains for the wealthiest serve to reinforce some economists' views that inequality in the U.S. is getting worse. Economist Thomas Piketty recently told CBS MoneyWatch that the gap between rich and poor has reached "spectacular" heights.