Noah Berlatsky is the author of " Nazi Dreams: Films about Fascism. " The views expressed here are solely the author's. View more opinion articles on CNN.

(CNN) By protecting the free speech of Nazis, we protect the free speech of all of us. That's the argument that the ACLU has put forward consistently , whether it's defending the right of Nazis to march through Skokie , or the Unite the Right rally in Charlottesville in 2017 .

Free speech is, according to this theory, a boat we're all in together. If you knock out the bottom to sink the KKK, we all go down.

A recent incident in Wisconsin, though, is an unsettling reminder that free speech for white supremacy doesn't necessarily have the intended effect: protecting other kinds of expression. A photo surfaced in mid-November of Baraboo High School boys giving an apparent Nazi salute before their junior prom on the Sauk County Courthouse steps. The Wisconsin school board recently ruled that they could not discipline the students involved because their salute was speech protected by the First Amendment.

Free speech purists claim that Nazi speech must be defended because it is especially controversial, and protecting it therefore also protects less controversial expression. But the school reaction to the Baraboo photo suggests this is not true.

Speech by white people is often seen as unobjectionable, no matter what its content; who you are is more important than what you say. Defending the speech of white kids doesn't necessarily protect the speech of marginalized people, just as All Lives Matter in a racist society does not in fact mean that Black Lives Matter. Racism is built on inequality and disproportion; you can't confront it by pretending that we're all in this together when we manifestly are not.

In fact, in practice free speech for Nazis is often itself a threat to free speech for everyone else, because Nazis use their freedom to violently suppress their opponents. Giving free speech to fascists to rally can reasonably be expected to curtail the free speech rights of other people, which means that organizations like the ACLU, and judges, need to balance interests, rather than just treating free speech for fascists as in itself increasing free speech.

James Alex Fields goes on trial this week for allegedly using a car to murder Heather Heyer, a woman peacefully counter-protesting against the Unite the Right rally in Charlottesville in 2017. White supremacists were given the freedom to organize and rally, and they used that freedom to terrorize people of color and others who disagreed with them. Fields, in particular, used his freedom to make sure that Heather Heyer would never speak again. (John Hill, an attorney for Fields, told prospective jurors the trial will include evidence that Fields believed he acted in self-defense.

In fact, "free expression" was just one of a blizzard of deflections offered by school officials to avoid having to discipline, or even censure, the Wisconsin students in the photo. District administrator Lori Mueller mused that "we cannot know the intentions in the hearts of those who were involved." Peter Gust, the photographer, says that he was asking students to "wave goodbye,"and said that no one realized how the picture would look.

Jordan Blue, one of the students in the picture, who refused to raise his hand, though, disputed that : "As soon as I heard the photographer say 'Raise your hand,' I knew what was going to happen." He knew how the picture looked, and he said other parents at the time were disturbed and objected to the photo.

Perhaps Gust was just confused, but the mix of explanations ("it's free speech"; "they were just waving goodbye"; "who knows what's in their hearts") suggests that the issue is less any one principle, and more a general desire to exculpate students. Black students apparently need to be disciplined rigorously and constantly in schools across the US, but white students are good kids who deserve protection, even when (especially when?) they make the sign for white power.

The school could have mandated detention or banned students from school activities for a time. Gust should be reprimanded by the school. The school has said it's going to put in place " restorative practices ," which is good. But if there's no sanction of any kind, how seriously will students view those efforts?

The point is not to inflict some sort of draconian punishment but to take steps to show that the school and school district take racism and bigotry seriously and stand in solidarity with Jewish students or students of color or others who might reasonably see themselves as targets in a school where Nazi salutes appear to be acceptable. But instead of taking a stand, they worked to find reasons to do nothing.

There's also a strong case to be made for vigorous free speech protections for students. In practice, though, nationwide, that case has gotten little traction when it is used to defend other kinds of speech. Any student knows that standing up in History and saying "this class sucks," won't be treated as a constitutionally protected act. Schools regularly ban clothing deemed to be too revealing, or that show anything school officials consider to be gang-related .

On social media, many defenders of the Baraboo boys have insisted that they couldn't be disciplined because they were off campus. But other schools have disciplined students for off campus conduct. One Oregon school suspended 20 students for a tweet saying that a female teacher flirted with her students.

The Supreme Court has in fact given schools explicit authority over student off-campus activities in the 2007 decision Morse v. Frederick . In that case, the court ruled that a school was within its constitutional rights to suspend a student who stood across from school grounds with a sign reading "BONG HITS 4 JESUS."

Morse v. Frederick is a terrible decision giving school authorities arbitrary and unconstitutional power to suppress speech. But it is currently the law of the land, which means that school officials have broad latitude to suppress and punish student speech and expression, on campus and off.

Inevitably, schools use that disciplinary authority disproportionately against marginalized students. Black students in 2013-14 were 15.5% of the US student population, but accounted for almost 39% of suspensions , according to data from the Government Accountability Office (GAO) released this year. Black students were also disproportionately subject to corporal punishment, expulsion, and school-related arrest.

In that context, the Wisconsin school board's reference to the First Amendment sounds less like a principled stance and more like an excuse. Free-speech rights are important, and school officials are given far too much power to silence their students. But if you truly care about free speech, it's important to acknowledge the ways in which free speech arguments can be leveraged to protect the powerful while others are silenced.

Fighting for the rights of the powerful to express white supremacy won't protect students of color who wear graffiti-patterned backpacks or doodle on their desks . White students making Nazi salutes, either apparently or intentionally, is perfectly consistent with black students being mercilessly policed.

For more opinion... Sign up for our new newsletter. Join us on Twitter and Facebook

A Nazi salute is a threat. When people start making them freely, the speech, and the lives, of non-Nazis are endangered. In a society with a racist history and a racist present, "free speech for all" in practice will very often mean free speech for white people and silence for everyone else.