It's easy to think that the stock market is the playground of hedge funds and day traders, but in reality most of the stock market is owned by the average joe.

In fact, the largest chunk is doing one thing: helping people retire.

In a white paper, Steven Rosenthal and Lydia Austin of the Tax Policy Center have broken out exactly which kind of investors own the stock market. They found that a majority of corporate stock is owned by different types of retirement plans, the largest being IRAs and defined-benefit plans.

Of the $22.8 trillion in stock outstanding (not including US ownership of foreign stock and stock owned by "pass-through entities" such as exchange-traded funds), retirement accounts owned roughly 37%, the most of any type of holder.

Rosenthal and Austin's main focus was the precipitous decline of taxable investment accounts. In 50 years, the amount of stock owned by individual investors and funds outside retirement and nontaxable accounts such as 529 college-savings plans has dropped off a cliff — to about 25% in 2015 from over 80% in 1965.

"After adjusting the data in several important respects, we estimated that taxable accounts held only 24.2 percent of C corporation equity in taxable accounts in 2015," the researchers said. "Our exercise revealed that the share of U.S. stocks held by taxable accounts declined sharply over the last 50 years, by more than two-thirds."

The other startling finding was the growth in foreign investment in the US stock market. What was once a small sliver of the makeup now accounts for a quarter of all stock ownership at $5.5 trillion. Part of this may be due to increasing wealth in foreign countries, but, as the researchers noted, it could also be influenced by corporate inversions, in which foreign-domiciled firms have large direct holdings of US-based stock.

Other significant chunks were owned by insurance companies ($1.6 trillion, or 5.6%) and nonprofits ($956 billion, or 4.2%).