Antifa, Trevor Noah pointed out on Thursday’s Daily Show, might have a cool name—”it’s short, it’s punchy (excuse the pun) and most importantly, you don’t have to know how to spell fascist”—but it also has a few problems. Although insistence from some camps on the right that the left-wing group antifa is just as bad as the fascists they fight is patently absurd, the anti-fascist movement’s lack of centralized leadership means that there’s no clear way to know what tactics it does and does not condone. Some members limit themselves to exposing and shaming Nazis online, while its more violent fringe takes things a step further, even breaking windows and calling it a “symbolic” act. That, Noah said, could cause the movement trouble in the long run.

Although Disrupt J20’s Lacy MacAuley said that acts like breaking a Starbucks window are “symbolic,” Noah pointed out that not everyone will see it that way. “Most people see that and think, ‘Great. Now I’ve got to walk a whole block up to the next Starbucks,’“ Noah quipped before getting more serious. “Who is that supposed to convince? What are you trying to do? It definitely doesn’t convince the Starbucks guy—the person who’s going to be cleaning up the glass [isn’t] saying ‘Yeah they made a good point about fascism.’“

As trivial as it might seem, optics are important--even when the cause is as dire as fighting fascists. And to Noah, a movement that includes violent or destructive fringes is at risk of being labeled dangerous. Indeed, the president has already tried to shift the blame for the violence in Charlottesville, Virginia, onto the left. Any violence, Noah argued, makes it easier for him and his supporters to do that.

“You see, now here’s the real problem: it doesn’t matter what your noble goal may be. It doesn’t matter what you say you’re fighting for. When people see that, all they think is, ‘Oh, shit, it’s vegan ISIS.’ . . . When you think you’re punching Nazis, you don’t realize that you’re also punching your cause. Because your opponents, they’ll just use every violent incident to discredit your entire movement. And they make it seem like in a world where white supremacists have a friend in the White House, the real problem is you guys.”