Many Americans have been rightly horrified by the videos of white supremacists shouting “Jews will not replace us!” But what has gone less noticed is that, unlike far-right American movements of years past, you don’t hear much talk of God or religion at today’s alt-right rallies.

Richard Spencer, the white supremacist and movement figurehead who coined the term " alt-right," discussed his atheism last year in an interview with atheist blogger David McAfee. When he posted the interview on his own website, Spencer retitled it “The Alt Right and Secular Humanism,” leaving no doubt that he sees atheism and humanism as linked to his cause. Yet I don’t know of any prominent atheist, humanist, or secular organizations that took the opportunity to condemn Spencer.

As someone who has worked in the atheist movement for the better part of a decade, this silence scares me.

I became an atheist when I was studying religion at a Christian college, and for years I felt very alone in my atheism. So I was happy to find an online atheist community in 2009 when I started a blog that eventually led to the publication of a book that detailed my journey to atheism. For years I spoke at atheist conferences and worked as a full-time humanist community organizer.

I’m still an activist, but after nearly a decade of active participation in online atheism (a loose community of forums, blogs, YouTube channels, and fandoms of figures like evolutionary biologist Richard Dawkins and writer Sam Harris), I mostly stepped away from the online side of atheism a few years ago. One of the biggest reasons for this was my growing concern over its failure to adequately address some of its darker currents—such as overt sexism, racism, and anti-Muslim bias.

Countless people I’ve spoken with over the years describe finding the movement through atheism’s frequently trafficked blogs and forums, just as I did. And there are of course valuable aspects of atheism’s strong online presence. Atheists who aren’t open about their beliefs—especially those living in totalitarian or ultraconservative environments where it isn’t safe to be open—can find resources that help them connect with likeminded peers, or simply feel less alone. Online forums and organizations like the Clergy Project, which offers anonymous support to religious clergy who no longer believe, positively impact people’s lives.

But there’s a toxic side to internet atheism. For years, women and people of color have repeatedly voiced how atheist websites, organizations, and public figures ignore their concerns and tolerate—or even actively contribute to—an environment that makes them feel unsafe and unwelcome, particularly online.

Given these concerns, it’s not surprising that areas of online atheism increasingly seem to be overlapping with the alt-right.

As George Hawley, author of Making Sense of the Alt-Right, told NPR last year, the alt-right is not only “predominantly white millennial men” but also probably represents “a more secular population than the country overall,” meaning many of its members are “agnostics and atheists or people who are just generally indifferent to religion.” Cultural conservatives are leaving organized religion, Peter Beinart argued in the Atlantic last year, and many are making their way into the darker fringes of the right.

Like the alt-right, American atheists—a growing segment of the US—are more likely to be male, white, and younger than the general population. Atheists are also one of America’s most negatively viewed groups and can face social isolation or family rejection. While religious people have churches, mosques, and synagogues staffed with care providers to help them connect with others, reflect on their lives, and find support in times of need, nonreligious people generally don’t have access to these kinds of resources. The alt-right intentionally targets and preys on people—young white men in particular—who feel disconnected, marginalized, and misunderstood, seeking to give them a sense of identity, belonging, and purpose. It’s not surprising then that atheists, who are often marginalized in America, may be prime targets.

By neglecting to address its darker currents, online atheism has perhaps unknowingly planted the seeds for the alt-right’s harvest. Three years ago Reddit’s atheism subforum, perhaps the largest community of atheists on the internet, was found to be the website’s third most bigoted—meaning not just tolerant of overt displays of bigotry, but actively supportive of them. Last year, the Daily Beast revealed that the study’s most bigoted Reddit subforum, the Red Pill, was founded by Robert Fisher, a Republican state lawmaker who is also an atheist.

The problem is more widespread than figures like Spencer and Fisher, too. While championing liberal views on some issues, many of atheism’s most prominent advocates—the majority of whom are, like me, cisgender white men—have expressed troubling sentiments that align with views held by the alt-right and faced little to no consequences.

Spencer’s views are not shared by the majority of atheists in America, but if we want to keep it that way, atheists cannot stay silent.

While there are certainly atheists and humanists doing the work and speaking out, too often atheists simply ignore this toxicity. And the silence of atheist organizations and public figures doesn’t come from ignorance. I spoke with a staffer at one of America’s largest secular organizations on the condition of anonymity who told me that issuing a statement condemning Spencer was discussed but shot down, in part because the organization’s leadership didn’t want to draw attention to the fact that Spencer is an atheist. This same organization, which regularly issues statements about political issues as a major part of its advocacy strategy, also reportedly declined staff requests to release a statement condemning Trump’s appointment of Steve Bannon as a White House adviser. According to the member I spoke to, the organization’s management didn’t want to publicly oppose anyone in Trump’s cabinet or inner circle.

When I asked my source why they thought a secular organization’s leadership would make these decisions, they replied, “When advancing atheism is your primary motive, you may have an interest in ignoring that some atheists are white nationalists and neo-Nazis. But staying silent also means keeping supporters who may otherwise be pissed off that you criticized the alt-right.”

As more and more nonreligious young white men seek out community, resources, and a sense of identity and purpose online, any overlap between online atheism and the alt-right should move atheists to speak out. Spencer’s views are not shared by the majority of atheists in America, but if we want to keep it that way, atheists cannot stay silent.

Some have argued that the mere fact of being an atheist does not obligate one to denounce Spencer, because atheism is not a belief system and Spencer is not a figure within the atheist movement, so his position in relation to the average atheist is different from the position of an average Christian in relation to a bigoted Christian leader. But when I discussed this with James Croft—a humanist community organizer who has also been deeply engaged in movement and online atheism for many years as a speaker and blogger—he emphasized that Spencer’s atheism is a central component of his worldview.

Croft pointed me to a second McAfee interview where Spencer suggested that he rejects Abrahamic monotheism because it says “we are all one,” and Spencer believes that civilizations need to define themselves in opposition to an “other.” So his atheism isn’t incidental; Spencer’s rejection of unifying religious messages is essential to his narrative of competing civilizations. Atheists who do not explicitly disavow this brand of atheism aren’t just missing an important opportunity to distinguish our community from Spencer’s dehumanizing ideas and actions. They are also failing to show that atheism does not necessarily lead to an oppositional attitude between peoples. In condemning Spencer, atheists have the chance to offer a robust, humanistic alternative to alt-right atheism that affirms the worth and dignity of all people to an increasingly secular generation.

But condemning Spencer and promoting an alternative aren’t enough. Atheists also need to ask ourselves difficult questions about the culture of our movement. Many atheists consider themselves transgressors who openly doubt and sometimes even mock the sincerely held beliefs of others—who take it upon themselves to slay “sacred cows.” This attitude is deeply embedded in movement atheism, where the most visible advocates tend to be vocally anti-religious. A 2013 study from the University of Tennessee at Chattanooga found that the atheists who consider themselves “anti-theists,” or vehemently opposed to religion in all its forms and eager to proactively fight it, have the highest rates of dogmatism and anger.

Croft suggested that this may be at the heart of the seeming kinship between so-called anti-theists and the alt-right. The taboo-confronting ethos of both movements, where irreverence is idealized and often weaponized, enables some of their members to style themselves as oppressed outsiders—despite often being relatively privileged straight white men. Many in the alt-right and atheist movements seem to see themselves as a group under siege, the last defenders of unfettered inquiry and absolute freedom of thought and speech, contrarians and truth-tellers who are unafraid to push back against the norms of polite, liberal society. If this is a part of why the alt-right seems to appeal to some atheists—and I suspect it is—then we must take a hard look at why that is and how to address it.

Many nonbelievers champion the idea that people can be “good without a god,” and while they’re certainly correct, it’s not enough to just say it. We need to live it. Otherwise it’s a slogan, just empty words. If there truly is no god that will fix humanity’s problems, atheist leaders and organizations that won’t take a strong stand against white supremacists don’t inspire much faith in our ability to do so ourselves.

The difficult truth spotlighted by both Spencer’s atheism and the silence of other atheists is that, despite the late Christopher Hitchens’s infamous proclamation that “religion poisons everything,” religion was never the problem. It was always something more complicated. Something uglier, more primal, more deeply human. Something the internet, with all the good it can foster, often facilitates. Until atheists and humanists confront this Something head on, we will continue to struggle with people like Spencer who embody an atheism that got rid of the gods but put white men in their place.

Sign up for our newsletter to get the best of VICE delivered to your inbox daily.