According to new data from the IRS, people who make $1 million or more had an average tax rate of 20.4 percent in 2010. Tax filers who earned $30,000 to $50,000 paid an average rate of 4.8 percent, while those who made between $50,000 and $100,000 paid 7.7 percent. Those making under $30,000 had a negative effective rate, meaning they paid no federal income taxes after deductions and credits.

Put another way, millionaires pay a rate that's more than four times that of the middle class.

One caveat: Rates go up as income goes up — but only to a point. Once you hit a certain magic number among super-high earners, your tax rates start to fall slightly.

According to the IRS, average tax rates increase as income increases — until you get to around $1.5 million in annual income. Once you make $2 million, average tax rates start to decrease. The average tax rate peaks at 25.1 percent for those making between $1.5 million and $2 million.

After that it starts to go down, and falls to 20.7 percent for those making $10 million or more.

So the millionaires who pay the highest average tax rates in America are those who make between $1.5 million and $2 million. That $2 million could be called the "Top Turning Point" on the income ladder, where rates reverse. (Related: Where the 1 Percent Live)

The reasons for this aren't complicated. Once you get above $2 million, your share of income from investments increases. Investments are generally taxed at the 15 percent capital-gains rate, compared with the top ordinary-income rate of 35 percent.

Those making $10 million or more earned nearly half of their income from capital gains and dividends.

Rates don't fall all that much once you get above $10 million. Even among the top 400 earners in America, whose average income is more than $200 million, the average rate is 18 percent — still more than three times the rate paid by the middle class.

Both sides of the current political debate on taxes will no doubt see these data differently. The right will say that the rich already pay more than their fair share, while the left will point to the low rates paid by the wealthy relative to the official tax rates.

But the figures show that the more you make, the more you pay — up to the Top Turning Point.

-By CNBC's Robert Frank

Follow Robert Frank on Twitter: @robtfrank