Credit the growing use of 401Ks and other retirement accounts for that phenomenon.

"Household equity holdings have more than tripled as a share of disposable income at the aggregate level since the late 80s," Goldman Sachs senior economist Daan Struyven said in a study Tuesday. "A 1 percent move in stock prices now has a much larger impact on wealth levels for most groups."

The wealthiest Americans are still the hardest hit by stocks – but now, more than ever, the rest of the country feels the pinch of a decline in the market.

Yet while the middle class own more stocks, their holdings still pale in comparison to how much stock the rich own. And their ownership of the equity pie has only gotten bigger. The wealthiest 1 percent now own half of all total household equities. They owned just 39 percent in the late 80s.

Because of that concentration, Goldman Sachs found that a shift in stock prices hits the wealthiest 10 percent of Americans three times harder what it did in the 80s. For those in the 50th to 90th percentiles, the hit is "a third larger," Struyven said.

Some have argued that the so-called wealth effect has reduced for the stock market over the years because equity holdings are so concentrated with those with a high net worth. While Goldman's work does show an increasing wealth gap, they believe that hasn't hurt the stock market's influence especially with stocks held by more individuals.

"The wealth effect from the stock market is unlikely to have fallen substantially over the past 30 years despite the rise in wealth inequality because equity holdings have risen sharply and because equity price moves still have a meaningful effect on the spending of wealthy households," Struyven said.

Declines in stock prices due still have "a meaningful effect on the spending of wealthy households," Struyven added. Goldman Sachs' data shows that "spending on luxury goods largely purchased by wealthy households is highly sensitive to stock prices," he said.