What would American authoritarianism look like? How will we recognize the collapse of democracy, and what would life be like under the new regime? And—perhaps most chillingly—would authoritarianism be equally oppressive for everyone? Or might we fail to notice when creeping fascism comes because it come creeping for someone else?

These questions have taken on renewed urgency this week as we move into the second year of Donald Trump’s presidency. It’s true that Trump did not discuss the Nunes memo or launch an attack on the FBI in his State of the Union speech last night. But Congress’ decision to release the document, which claims, without evidence that the FBI has been plotting to undermine Trump, is chilling. House Speaker Paul Ryan has said that it is time to “cleanse” the FBI—which in practice means replacing personnel with Trump loyalists. If that happens, and Trump in 2020 demands an investigation of the Democratic candidate, who would stop him?

“It’s a mistake to think a dictatorship feels intrinsically different on a day-to-day basis than a democracy does,” writer G. Willow Wilson noted in a thread on Twitter. Authoritarianism doesn’t necessarily slam down all at once. It often is a slow erosion of norms—a law enforcement agency corrupted here, the self-censorship of media there. It can sneak up on you. And that’s especially true because authoritarianism is not egalitarian. It doesn’t affect everyone in the same way.

In most popular dystopian fiction, the oppressive regimes oppress with an admirable lack of discrimination. There’s a powerful elite, and that elite comes for all the people at once. In Star Wars, the Empire, or later the First Order, obliterates entire planetfulls of enemies; it destroys anyone who gets in its way, whether heroic human rebels, droids, or fish-faced aliens. When Loki tries to conquer the earth in The Avengers, it’s the entire earth he wants to conquer; everybody, of every class, race, and gender is forced to kneel to him together. Orwell’s 1984 goes even further. The most oppressed people under Big Brother are Party members; the proles have more latitude and more freedom. In our single most cited dystopia, oppression is topsy-turvy—marginalized people are improbably, the least policed.

Based on history, and for that matter based on the present, these uniform dystopias are very misleading. Authoritarian regimes almost always do discriminate; they consolidate power by demonizing and scapegoating marginalized or despised groups.

Hitler targeted Jews and Communists. Stalin’s worst atrocities were perpetrated not against citizens at home, but against Ukrainians and Poles. Many gentile Germans under the Nazis and Russians under Stalin were certainly oppressed. The dictators were eager to torture, exile, and murder any of their own people who criticized their regimes. But, nonetheless, certain groups were much more likely to feel the boot of the state on their necks than others. “First they came for the Socialists, and I did not speak out—Because I was not a Socialist,” protestant pastor Martin Niemöller declared. Niemöller meant that statementas a warning, but it’s also a reminder that totalitarians tend to have a hierarchy of oppression. They come for some people first.

In the United States, the totalitarians tend to come first for black people and people of color. It’s not difficult to imagine what authoritarianism would look like in the United States, because it already happened here. The Southern United States, before the Civil War, was a gulag, in which almost 4 million people labored under a vicious authoritarian regime. Enslaved people were beaten and had fingers and toes amputated for daring to learn to read—Ray Bradbury’s Fahrenheit 451, like so many other dystopias, basically just involved applying the historical conditions of marginalized people to white folks. The same could be said of Margaret Atwood’s Handmaid’s Tale, in which the horror is that a college-educated white woman experiences the kind of systemic, sanctioned rape and sexual violence which were regularly visited upon black women under slavery.

American democracy has always existed alongside American totalitarianism; liberty for some sits comfortably beside authoritarian violence for others. Politicians often refer to the United States as the freest country in the world, even as we have the largest prison population, by numbers and percentage, of any country on earth. But prisoners are disproportionately black and Hispanic, as well as disproportionately poor, so our police state isn’t seen as a police state. But can you credibly claim to be a free people when you literally cage more than 2 million people?

Similarly, the most authoritarian aspects of Trump’s regime are often ignored because they are targeted rather than universal. Trump’s tentative efforts to undermine the FBI are disturbing. But his administration’s assault on immigrants are a lot more so. The Immigration and Customs Enforcement Agency (ICE) under Trump has staked out hospitals to arrest 10-year-olds with cerebal palsy and appears to have altered documents in order to justify deporting a man with a legal work permit. A GOP congressman this week threatened to have immigrant guests of Democratic congresspeople arrested at the State of the Union speech. ICE aggressive actions, and anti-immigration rhetoric, have created a climate of terror in immigrant communities—people are afraid to access health services, send their children to school, or report crimes to the police.

Totalitarian regimes, of course, thrive on fear. Draconian, arbitrary state violence is intended to prevent people from speaking, and to drive them away from the political sphere—so that, for instance, people who want to protest immigrant policies are afraid to show up at the State of the Union speech for fear of arrest. We don’t need to imagine what widespread surveillance and state intimidation would look like; it’s already here. It’s just that the surveillance and the intimidation are directed at groups like immigrants, black people, and prisoners, and so most Americans don’t feel directly threatened.

Trump, many fear, is undermining our traditions of democracy. But just as dangerous is the way that he is expanding on and extending our traditions of totalitarianism—traditions we find it dangerously easy to ignore and deny. American authoritarianism probably won’t involve a revolution or a coup. Probably, it will simply mean that marginalized people will face more state violence, and ever diminishing freedom. Middle-class white people will perhaps be mildly inconvenienced in the name of greater “security”, but on the whole won’t notice much difference in their day to day lives. That’s how American authoritarianism has worked in the past, that’s most likely to be how Trump’s authoritarianism works in the future. The United States has long known that totalitarianism works best when it’s unevenly distributed.