Checking my Facebook feed on Friday night, I came across a chilling image: A University of Virginia law student had posted live video of white supremacists marching with lighted torches through her darkened campus. To anyone who immerses herself in the history of the 1930s and 1940s as I do, the video was reminiscent (intentionally so) of nationalist students’ torch-lit parades through university towns at the beginning of the Nazi era.

Of course it only got worse in Charlottesville the next day as the white supremacist, neo-Nazi protests turned first violent, then fatal. Since then, we’ve all been reminded of the parallels to the earlier fascist era and alerted to the dangers of allowing small, extremist groups to grow into movements that overtake civilized society.

>>Charlottesville proves Confederate monuments embody heritage of hate ■ What I discovered about white supremacists while protesting alongside rabbis in Charlottesville ■ Can white males be terrorists in Trump's America?

For someone who studies the American response to the Holocaust, however, another lesson has emerged from the disturbing events of last weekend. In the 1930s, Americans didn’t just watch Hitler from afar. Inspired by the fuhrer’s rhetoric and the supposed utopia he was building in Germany, some Americans formed their own pro-Nazi, pro-fascist and anti-Semitic, anti-foreign groups. They bore scary names such as the Silver Shirts, the German American Bund, the Defenders of the Christian Faith and the Knights of the White Camelia. A revitalized Ku Klux Klan included virulent antisemitism in its hateful repertoire. In 1938, these groups even formed a united organization, the Christian Front, that held rallies calling for Jews in America to be liquidated and ending with the Nazi salute. In February 1939, 19,000 Christian Fronters and other Nazi sympathizers gathered in Madison Square Garden to denounce the Jewish domination of Christian America and to shout ‘heil’ in support of their foreign leader.

For all their storm and fury, however, these groups never drew large numbers of followers. William Dudley Pelley’s Silver Shirts had at most 25,000 members in a nation of 130 million people. Gerald Winograd’s Defenders of the Christian Faith had perhaps 100,000 readers of its newspaper, but no actual organization to defend the faith. Only Father Charles Coughlin, the Catholic priest who formed the Christian Front, drew a large audience for his anti-Semitic radio show. Once World War II started in September 1939, Coughlin’s pro-neutrality stance became too much for the Catholic hierarchy and he was forced off the air.

Open gallery view In this May 22, 1938 photo, provided by the New York City Municipal Archives, members of the German American Bund pose for a photo at Camp Siegfried, in Yaphank, N.Y. Credit: /AP

Yet, these American Nazis still managed to have a profound impact on American society. They did it not by representing an influential populist movement, but by convincing nervous politicians they were enough of a force that xenophobic and anti-immigrant policies should prevail. Of course, many government officials didn’t need any encouragement to be racist and xenophobic. The nation had already taken a nativist turn in the 1920s, ending what had in effect been unrestricted immigration. The Congress imposed a 150,000-limit on annual overall immigration and country-by-country quotas that, for Germany, amounted to just 25,000 immigrants a year.

Still, the thugs marching in the street and gathering in the Garden in the 1930s heightened the sense that any softening of the nation’s immigration posture would result in political retribution at least and violence at worse. The American Nazis intimidated some lawmakers, emboldened others, and gave cowardly ones a ready explanation for their restrictive policies. The anti-immigrant atmosphere reassured State Department officials that they could continue to use administrative measures to keep immigration way below even the legally allowed limits without concerns about a congressional or public outcry. And the American Nazis contributed mightily to the trepidation within the American-Jewish community about advocating on behalf of their European brethren. As a result, American hate groups in the 1930s had influence on American policy way out of proportion to their actual presence within American society.

That is a lesson we need to take to heart today. We need to be careful not to exaggerate contemporary white supremacists' and neo-Nazis' power. That doesn't mean we should tolerate their hateful rhetoric any more than their hateful actions. It doesn't mean that we should accept the president's absurd false equivalence between hate groups and those trying to counter them. But it does mean that we need to speak and act from a position of strength. We represent the overwhelming majority of Americans. Even most Trump backers, including those who harbor racist sentiments, don’t support shouting ‘sieg heil’ and calling for race war.

Americans’ experience with homegrown fascism in the 1930s shows that so far at least, we have more to fear from the fear of extremist groups than the groups themselves.

Laurel Leff is Associate Director of Jewish Studies and an Associate Professor of Journalism at Northeastern University in Boston. She is the author of "Buried by The Times: The Holocaust and America's Most Important Newspaper", and is currently researching the American response to the refugee crisis of the 1930s and 1940s.