“I disapprove of what you say, but I will defend to the death your right to say it.”

The quotation—incorrectly attributed to the French enlightenment writer Voltaire—sums up the American ideal of free speech. The basic idea is that, in order for freedom to flourish, people of good will must protect even repulsive speech—up to and including pornography, racism, sexism, bigotry, and in some cases, generalized calls to violence. Free speech must be universal, the argument goes. If Nazis are not able to speak, we will all be silenced.

This principle was sorely tested over the weekend in Charlottesville, Virginia. Nazis were permitted to march and speak. The result was not more freedom for all. Instead, the march ended, predictably, in horrific violence. One of the people attending the white supremacist march drove his car into a crowd of peaceful counter-protestors, killing a woman named Heather Heyer and seriously wounding many others. Letting Nazis congregate didn’t allow others to speak; it silenced at least one person forever. Defending fascists’ right to speak their minds resulted in the death of someone else. The violence in Charlottesville bleakly suggests that free speech absolutism—without anti-fascism—leads to less free speech for all, not more.

Free speech defenders vigorously reject the suggestion that, as an ideology, free speech absolutism may fail in some situations. The American Civil Liberties Union has a long history of defending neo-Nazis’ right to hold marches and rallies. In line with that tradition, the ACLU of Virginia came to the defense of Unite the Right organizer Jason Kessler and prevented the city of Charlottesville from moving the site of the rally from Emancipation Park, despite the city’s safety concerns. The ACLU’s legal position prompted a board member to resign. It also led many on social media to suggest that the ACLU had paved the way for fascist violence.

Constitutional lawyer and Intercept founder Glenn Greenwald responded by reiterating the tenets of free speech absolutism in his usual polemical style. “Purporting to oppose fascism by allowing the state to ban views it opposes is like purporting to oppose human rights abuses by mandating the torture of all prisoners,” he declared. Fascism believes in suppressing free speech, he argued; therefore suppressing free speech of Nazis is actually cosigning fascism. Courts rely on legal precedents, Greenwald says. If the ACLU had failed to stand up for neo-Nazis protesting in Charlottesville, the next time marginalized people wanted to march, they could be silenced by the state. “We defend the rights of those with views we hate in order to strengthen our defense of the rights of those who are most marginalized and vulnerable in society.”

This is certainly a logical and coherent argument. But logical and coherent arguments don’t always pan out in practice. Does defending the right of people to spout hateful views consistently protect the marginalized? Writer and activist Julia Serano points out in a Medium post that as a young adult, she could not tell people she was trans because of the likelihood that she would be greeted with freely expressed bigotry and hate. “Of course, I technically had free speech, but that doesn’t count for much if speaking your mind is likely to result in you being bombarded with epithets, losing your job, being ostracized by your community, and possibly other forms of retribution,” she writes. Any unmoderated comments thread on the internet provides similar evidence that free speech for all often means silence for a few. Hateful, bigoted speech, if left unchecked, leaves marginalized people feeling vulnerable and endangered—for good reason. If you let people spew bile, the folks at whom they spew bile will leave. You’ll be left with a safe space for hateful speech in which the only speech on offer is hate.

Free speech absolutism also elides the issue of race. Neo-Nazis may be expressing hated views, but they are still white, and law enforcement, the courts, and the state will treat them accordingly. In Ferguson in 2014, mostly black anti-racist protestors were met with an overwhelmingly militarized response; 155 people were arrested. In Charlottesville, by contrast, despite numerous incidents of violence, police arrested only four people.

Defending free speech rights absent a specific commitment to anti-biogtry and anti-racism is meaningless. Mariame Kaba, founder of Project NIA and an anti-prison activist, noted on Twitter that “these ‘convos’ about civil liberties are completely divorced from the realities of living Black in the U.S….Civil liberties and individual rights have different meanings for different groups of people.” In a context where black people are denied basic rights and freedom as a group, “black people have focused on our collective rights over our individual liberties….As a people, we’ve always known it is impossible for us to exercise our individual rights within a context of more generalized social, economic, and political oppression.” A supposedly color-blind approach to free speech just ends up reinforcing the status quo— whereby the state default is to arrest non-violent black people and lets violent white people walk free.

Internationally, it’s clear that free speech absolutism and defending Nazis is not the only option for people who want to create a just and free society. Germany uses anti-hate speech laws to prosecute people who make bigoted and xenophobic statements. These laws are sometimes used against other kinds of speech too; Germany is not a perfect utopian society. But non-Nazi protestors in the US regularly face draconian punishments as well. If the ACLU had decided not to support the right of Nazis to march wherever they wanted, regardless of safety threat, would the US really descend into (more of a) nightmare dystopia? I’m skeptical.

Free speech absolutism is a faith. Though people marshal pragmatic arguments on its behalf, the real argument is a moral one. The ACLU and Greenwald are committed to free speech for all because free speech is their most important ideal—it is the good thing from which equality, freedom, and all other good things flow.

For people who see themselves as anti-racists and anti-fascists first, however, the insistence that free speech will save us all rings somewhat hollow after this weekend. Given limited energy and resources, maybe defending the rights of violent bigots isn’t the noble choice in every case—especially when those bigots predictably use their platform to silence others. Free speech absolutists insist that free speech is the foundation of anti-fascism. But maybe anti-fascism is the basis of true free speech—in which case, defending the speech of bigots can, at least in some cases, leave us all less free.