The widening gap between the wealthy and the rest of America during the recovery can largely be explained in one word: stocks.

According to recent data from the Federal Reserve , America has the lowest level of stock ownership in 18 years. Yet stock ownership for the wealthy is at a new high-and that has accounted for most of their good fortune compared to the rest of America.

Read More Wealth gap 'unsustainable,' may worsen



The Federal Reserve Survey of Consumer Finance found that only 48.8 percent of Americans held stock either directly or indirectly in 2012, the latest period measured. That's the lowest level since 1995, when 40.5 percent of Americans held some form of stock. (Indirect ownership of stock includes stocks held in mutual funds, 401-K plans and other investment vehicles.)

The survey said only 14 percent of Americans own stocks directly-down from 21 percent in 2001.



But even more than most assets in America, their ownership is highly skewed toward the wealthy. Fully 93 percent of the wealthiest 10 percent of Americans own stocks. That's nearly twice the level for the middle 50 percent and far more than the 26 percent stock-ownership rate for the bottom 40 percent.



Read More Growing wealth gap among the rich



Stock ownership is even more concentrated when it comes to share of total stock holdings. In 2010, the latest period available, the top 10 percent of Americans by net worth held 81 percent of all directly held or indirectly held stocks, according to Edward N. Wolff, an economics professor at New York University who specializes in inequality and Federal Reserve data.

Wolff said that share-which has not been released yet for 2013-has probably gone even higher than 81 percent since 2010.

Read More Wealth gap widened over past decade: Report



"The stock ownership rate for Americans peaked in 2001," he said. "It's been going steadily downhill since then."

And that, he said, has widened the wealth gap between the top 10 percent and the rest of America.





The flip side of the data is that those who have stayed in stocks-whatever their wealth level-have generally done the best in the recovery. The mean value of stock holdings for those who held them hit $269,000 in 2013, up 18 percent from $228,000 in 2010 and up from $137,000 in 1995.

Among the top 10 percent, the mean value of stock holdings soared to $975,600 in 2013, up from $834,800 in 2010. For those whose net worth puts them in the bottom 20 percent to 40 percent of Americans, the mean value of their holdings rose to $51,000 from $44,000 in 2010.

Stocks have helped the rich get richer in this recovery. But they have also helped the less-rich get richer.

-By CNBC's Robert Frank

