Everyone knows the Top 1% have been killing it in recent decades, while most Americans have seen their incomes stagnate.

But what if the middle class were doing as well as the richest Americans? What would their annual income be?

CNNMoney asked the Economic Policy Institute, a left-leaning research group, to crunch the numbers.

The answer: $156,318.

Instead, the average household income of the middle class is less than half that: $72,036. That's up only 17% from 1979, when it was $61,542.

(EPI defines the middle class as those in the nation's 20% to 80% income range, and used data through 2011. It uses Congressional Budget Office data, which includes gains from investments, as well as certain public assistance and employer health insurance.)

The fortunes of the Top 1% and everyone else began to diverge in the late 1970s, when the richest Americans were earning $613,313. They have since seen their average household income skyrocket by nearly 150% to more than $1.5 million.

There are several reasons why the Top 1% has done so much better, says Josh Bivens, EPI's research and policy director. These include tax cuts for the rich, deregulation of Wall Street and big increases in executive compensation. At the same time, the weakening of unions and the failure of the minimum wage to keep up with inflation depressed the wages of the middle class.

"Intentional policy decisions in Washington and at the state level shifted bargaining power from workers to those at the top," Bivens said.

Since 2011, the Top 1% has likely continued to prosper, Bivens said. A good chunk of their income depends on the stock market, which has done very well in recent years.

Workers, on the other hand, have seen wages remain flat, despite the improving economy.