The House of Representatives passed its tax bill Thursday, making way for a sweeping $1.5 trillion tax cut. The showdown on taxes now heads to the Senate, which is set to vote on its bill the week after Thanksgiving.

The biggest impact of both plans is that America’s big corporations and wealthy would have to pay less in taxes because of new deductions and loopholes the bills would create. Meanwhile, poorer Americans and the middle class would actually see their taxes go up.

The House plan would get rid of the estate tax, which kicks in when a very wealthy person leaves assets worth $5.49 million or more to their heirs. The Senate plan doesn’t eliminate the estate tax entirely, but it would dramatically lower the number of people who have to pay it by doubling the threshold for taxable assets from $5.5 million to $11 million.

There’s plenty of polling to show that most Americans don’t think the wealthy deserve such steep tax cuts (though it’s also worth noting that throughout the years, there have been multiple polls showing that keeping the estate tax isn’t that popular). But the latest poll on the GOP tax plans shows a clear picture: Most Americans aren’t happy with the proposal that the middle class pay more taxes while wealthier Americans pay less.

Middle-class voters are in agreement with a surprising group of people: millionaires. Since 2010, a group called Patriotic Millionaires has fought for the closure of a number of tax loopholes and workarounds that allow people like them to pay less while the tax burden grows on working families.

I recently interviewed the chair of the board of Patriotic Millionaires, Morris Pearl, on why he sees income disparity growing in the United States and how he believes the House and Senate tax bills would add to that. Pearl previously served as the managing director of the investment firm BlackRock.

This conversation has been lightly edited for length and clarity.

Ella Nilsen

Why are you advocating for millionaires to be paying their fair share of taxes, or even be paying more taxes?

Morris Pearl

There’s a possibility of things getting much worse if we have even more disparity between the top people and the rest of the people, leading toward maybe even civil unrest. I don’t want to live in a country like apartheid South Africa or a banana republic or something, with a few rich people living behind walls and guarded compounds and being driving around in armored cars. I remember going on vacation to some small island someplace once with my kids, and it didn’t take long to realize we didn’t even quite enjoy that environment. And I don’t want the United States to be like that. Even if it’s for my own personal, selfish reasons. So that’s where I come from.

Ella Nilsen

Do you see the United States slipping into that?

Morris Pearl

Well, I see much more disparity between the people at the top and the people at the bottom. It’s the people at the top and most people, in fact. We have much less of a middle class than we used to; we have a small group of very wealthy people, we have a large group of very poor people, and they barely even meet each other.

Seeing it is becoming almost a bimodal kind of a thing, with a lot of people that are shuttled around in limousines to private airports and fly on private planes, and we see on the evening news about people waiting in line in TSA and that’s like reading about a different country.

So yeah, I see growing disparity between the top whatever and the rest, and I am afraid of it getting worse, and I think it is getting worse. I don’t think that’s good for anybody, really. Not for me, not for anyone else.

Ella Nilsen

I’m curious, having looked at the House and Senate tax bills, if you feel that’s going to add to the disparity currently in the country, and if so, how?

Morris Pearl

Well, yes. I of course don’t know every detail, but I’m particularly concerned about, for instance, the estate tax. For many of the wealthiest people in the country, that’s the only tax they ever pay. Look at the list of billionaires in Forbes. Eight of the top 20 are people who are heirs of either Sam Walton, who started Walmart, or Forrest Mars, who started Mars Candy. They built businesses from zero to essentially billions and billions of dollars by the time of their demise, those two people. And they never paid any tax on that huge increase in wealth.

Yeah, they paid a tax on the salaries that they drew over that over their lifetime. But they never paid any tax on that huge capital gain, and neither will their heirs; because of the step-up basis rules, they only pay tax on amount of gains after their patriarch died.

And so the estate tax — the only tax these people ever pay, because they don’t pay taxes — the value of their investment goes up, because it’s never realized. And essentially, if we have no estate tax and no tax on capital gains until realized, and step-up basis on inheritances, we’re going to have a small group of people who are just sort of the permanent oligarchy class, forever. And they’ll completely lose sight of the rest of the country.

And I don’t think that is what America was supposed to be. So I think getting rid of the estate tax is a really bad idea, not to mention it’s only paid by a quarter of 1 percent of the wealthiest people.

Ella Nilsen

You are a millionaire; other people in your organization are millionaires. Coming from this world, what are some of the things you think the rest of us are missing, with the reality of what wealthy people have to pay and don’t have to pay?

Morris Pearl

The reality is that taxes are paid on income, not paid on wealth. So you can be extremely wealthy and pay no taxes at all if you don’t have any income. You can be wealthy enough to be spending millions of dollars a year and not pay any taxes at all if you don’t have any income. Now, income includes selling things. But if, for instance, the people that inherited the Mars candy fortune just had to sell a little bit every year, if they’re selling it at the same price that it was worth when Forrest Mars Sr. died, they’ll never pay any income tax at all.

That’s the key thing — it’s not how rich you are, it’s how much income you have. So people that work for a living, they pay very high tax rates in this country: 39 percent if, say, you make $500,000 a year or so. People who inherited billions of dollars and spend a few million every year, they pay no taxes at all.

And the current proposals are making that even more unequal, because they’re having even less taxes on investment income and more taxes for those people that actually work for a living. They honestly believe that people who don’t work but invest their money are somehow better people and they shouldn’t be paying taxes. So that’s the key thing.

Ella Nilsen

Why do you think there aren’t more millionaires and billionaires that aren’t saying what you guys are saying, that aren’t saying, “We want to pay more taxes”?

Morris Pearl

I think a lot of people agree with us, but they don’t feel like stepping out in front because they’re afraid of other people thinking they’re communists or anti-capitalists or something.

A lot of people, frankly, are with us. Bill Gates, Warren Buffett, two of the wealthiest people in the country, are with us, and so are plenty of billionaires — Tom Steyer, George Soros. A lot of billionaires are with us on these things. All of those people built businesses depending on American consumers buying stuff.

The few people who are most opposed to us are people who built businesses based on extracting minerals from the earth and selling them. They don’t really care how much their workers make because they don’t plan to sell anything to their workers, and they have this misconception that they somehow deserve everything they get because they’re the ones who are just making money by selling stuff that God provided to them. Those are the people most against us, people who are doing coal mining and oil drilling and stuff like that.

Ella Nilsen

What’s been the message that you’ve gotten when you went to Capitol Hill and talked to lawmakers in the past?

Morris Pearl

Many are on our side, almost all Democrats. When we go to Republicans, the message is usually, “Oh, of course, we’re in favor of that as part of a long, comprehensive tax reform plan.” Typically, we’re talking about the carried interest tax loophole [a loophole for private equity managers, venture capitalists, and hedge fund managers]. Very few actually defend it.

A few people say they’re not in favor of increasing anyone’s taxes, ever, because they think the taxes are too high, and they want to decrease everyone’s taxes. Those who do respond, it’s usually, “This would be part of a comprehensive tax reform that lowers tax rates.” It depends on what you mean by “reform.” Most everyone who says they want tax reform means they want to pay less taxes.

Outside of our group talking about these issues, particularly the carried interest tax loophole, which we believe is sort of the poster child of egregious tax loopholes, I rarely hear of anyone saying they want any taxes raised.

Part of the problem with money in politics is that very few people are out there hiring lobbyists to talk about issues that are sort of for the common good, whereas some [representative] of [a] narrow interests like the home mortgage issuer association is able to hire some lobbyist to be able to talk about how important the home mortgage tax deduction is. But there’s very few people that are lobbying for fairness in general.

Most people don’t really care that much what other people’s taxes are. Most people are not really going to take a lot of effort to lobby for other people’s taxes to be raised. It doesn’t really affect them that much directly.