Income inequality has increased in recent decades: nobody disputes that. But there is plenty of argument over the main story. Is it the separation of the very rich in the top 1 percent of the distribution (the “we are the 99 percent” story), or is the fracture that counts lower down, with the top 20 percent pulling away from the rest?

I am in the latter camp. In my book Dream Hoarders, I argue that the obsessive focus on the top 1 percent has distracted from the relative success of a much bigger group, the upper middle class who make up the top quintile. This has generated plenty of dissent. Matthew Yglesias of Vox, for example, worries that my position is “exculpatory” of the top 1 percent. I don’t think so. In the book, I write as follows:

I am not suggesting that the top 1 percent should be left alone. They need to pay more tax, perhaps much more. WE need to find ways to stop them from moving money around and abroad in order to avoid paying, at minimum, what the current tax laws demand. (If there had been any doubt about that need, it evaporated in 2016 with the leak of the so-called Panama Papers.) But if we are serious about narrowing the gap between “the rich” and everybody else, we need a broader conception of what it means to be rich.

Of course, all lines drawn in the income distribution, whether at the 80th percentile or 99th percentile are somewhat arbitrary. In the real world, for example, people move in and out of the top 1 percent quite regularly. Still, an important empirical question remains. Where is the inequality fracture located? Let’s start by looking at income trends for the bottom 40 percent of households, the next 40 percent, and the top 20 percent:

This seems to support my story. (Well, I did choose the chart.) But of course, the top 20 percent includes the top 1 percent. It is perfectly possible that the whole quintile is being pulled up by spectacular income gains at the very top. So, let’s look at the top quintile both without out the top 1 percent:

What does this tell us? I think it tells us that the separation of the top 20 percent is more marked when the top 1 percent is included, but that it is clear even if they are excluded from the calculation. Between 1980 and 2013, average incomes in the bottom 40 percent grew by 39 percent and in the “middle” 40 percent income grew by an average of 36 percent. The income of the 20 percent including the top 1 percent rose by a whopping 92 percent. But even without the 1 percent, the rise was 70 percent, twice that for those in the middle. Breaking the data down further demonstrates this point. The higher up we go, the greater the separation:

So my critics are right to say that the further we go up the income distribution, the greater the observable income gains. But I think they are wrong to suggest that the income inequality story can be told solely in terms of the top 1 percent. When we turn to other dimensions of inequality, such as education, housing, family, health and so on. I think it makes even less sense to frame the divide in 99 percent v 1 percent terms. It is worth saying, however, that many of the policies that mostly benefit people in the top 20 percent are even more valuable to those in the top 5 percent, and really, really valuable to those in the top 1 percent. (Exclusionary zoning, legacy preferences in college admissions and tax deductions on mortgage interest and 529 college savings plans are good examples).

As I argued in my New York Times piece “Stop Pretending You’re Not Rich,” there is a growing class barrier between the upper middle class and the rest of America. Narrowing the debate to a focus on income alone, and to the top 1 percent, runs the risk of being exculpatory of the next 19 percent.

Some believe the “we are the 99 percent” framing is good politics. The last decade of political history suggests otherwise. Time for a tougher conversation.