In 2012, a gay couple in Colorado asked a baker to make them a wedding cake. The baker refused, saying that to do so would mean going against his religious beliefs. The Colorado government sided with the gay couple, arguing that the baker’s decision violated state anti-discrimination laws.

The case has made its way to the Supreme Court, which will hear Masterpiece Cakeshop vs. Colorado Civil Rights Commission on Tuesday. The case is being framed as a battle between free speech and equal rights. The Cakeshop owner argues that he should not have to bake cakes for same-sex weddings, because cake-baking constitutes expressive and artistic speech. Colorado anti-discrimination law, however, prevents public businesses from discriminating against individuals on the basis of race, religion, or sexual orientation. So far, Masterpiece Cakeshop has lost every court battle; the ACLU has weighed in against it as well.

If the Supreme Court rules in favor of Masterpiece Cakeshop, it will punch holes in a raft of anti-discrimination laws across the country. But this will not be a victory for freedom of speech. Rather, the decision will foreclose free speech—and make America a more restrictive and more totalitarian society.

America’s free speech traditions evolved in a country that embraced slavery, displaced native peoples, and denied women the vote. Free speech always meant freedom for certain people at the expense of others. Slave owners demanded the freedom to do what they would with their “property,” while enslaved people who complained of their plight in public could be beaten or murdered outright.

In that context, the “free speech” tradition has been oddly divorced from “freedom.” Or, to put it another way, free speech in the US has historically been applied only to people who were already considered equal and free. The ACLU argues that free speech helped defend members of the Civil Rights movement. But free speech protections certainly didn’t help Fred Hampton, the Black Panther leader murdered by the FBI for political speech and organizing. When US Immigration and Customs Enforcement targets undocumented people who speak out about immigration policy, the goal is to make clear to immigrants that they better keep quiet.

Free speech absolutists often imagine an open public square, in which everyone has access to the same podium, and can advocate for whatever they wish—whether that’s socialism, capitalism, or the extermination of certain racial groups. If all opinions are equally protected, all people are equally free. The job of the courts and the government, in this view, is to make sure no one is silenced for what they say or believe.

The problem is that, in practice, people are most often silenced not for what they believe but for who they are. Totalitarianism rarely looks like 1984, in which everyone is equally repressed, and the party members with more power are most policed. Rather, totalitarianism usually looks more like the Jim Crow South, in which one stigmatized group was terrorized in the name of preserving freedom for their oppressors.

This is very much true of the experience of LGBT people. Gay, lesbian and queer history is one of painfully enforced silence. In the US, freedom of speech protected the right of LGBT people to say anything, as long as they did not discuss the truth of their identities. This position was codified through the military’s “Don’t Ask, Don’t Tell” policy, which mandated that LGBT soldiers could stay in the military as long as they lied about their sexuality. This was a government policy that literally punished people for speech—because that speech was about their own marginalized identity. Though scholars argued that “Don’t Ask, Don’t Tell” violated the First Amendment, it was not struck down on those grounds.

If the Supreme Court allows Masterpiece Cakeshop to discriminate against LGBT people, it will be a step back into the closet—which is to say, it will be a step toward silencing gay, lesbian, and queer people in public spaces. If businesses can refuse service to LGBT people, then there is a powerful incentive for LGBT people to censor or silence themselves when trying to buy cakes—or do anything else.

What if you want to book a flight for your honeymoon, and the airline’s management decides they don’t approve of lesbian weddings? What if you want to buy a condom, and the convenience store owner doesn’t approve of gay sex? Perhaps these discriminatory actions wouldn’t hold up in court. But a ruling in favor of the Masterpiece Cakeshop would certainly empower some store owners to test the limits. A government that sanctions bigots empowers bigotry. And empowered bigotry forces the targets of bigotry to watch what they say.

Nor are LGBT people the only ones who could have their public speech restricted. If you don’t have to make cakes for LGBT people, what’s to prevent you from refusing to make them for Jewish people, or Muslims? If you can exercise your free speech to discriminate against one group, why not discriminate against others?

Members of marginalized groups sometimes do not know for sure when they enter a store whether or not they will be served. In 2017, it’s still possible that they will instead be refused, insulted, and publically humiliated. Masterpiece Cakeshop could make this possibility even more likely. Essentially, at worst, minorities will no longer be welcome in public unless they pretend to be something they are not. At the end of that slippery slope is a world in which heterosexual people can insult LGBT people as freely as they please, and LGBT people must retreat into silence.

Hopefully, the Supreme Court will decide against Masterpiece Cakeshop. But whatever it decides, it’s long past time we started to redefine freedom of speech. What matters is not just what people are allowed to say, but who is allowed to speak. Discrimination restricts where people can live, where they can go to school, what jobs they can get, and what sort of treatment they receive when they are stopped by police. But discrimination also, always, is meant to shut certain people up. Tyranny is predicated on the silencing of the marginalized. Without equality, there is no free speech.