This was originally supposed to be a Daily Dot piece (hence the un-blog-like format), but that didn’t work out. So instead, here’s a post in which I interviewed a bunch of cool atheists about how it is that we manage to have morals. Yay!

Phil Robertson, controversial star of A&E’s Duck Dynasty, recently chose a bizarre way to try to prove that atheists have no morals. He concocted a brutally violent rape scenario and shared it in a speech at a prayer breakfast:

Two guys break into an atheist’s home….they take his two daughters in front of him and rape both of them and then shoot ’em and they take his wife and then decapitate her head off in front of him. And they can look at him and say, ‘Isn’t it great that I don’t have to worry about being judged? Isn’t it great that there’s nothing wrong with this? There’s no right or wrong, now is it dude?’

While many have criticized Robertson’s graphic speech, plenty of his fans defended him, even starting an #isupportphil hashtag on Twitter. But whether you believe in god or not, there is nevertheless something bone-chilling about the apparent glee with which Robertson relays his story, especially considering that the group he targets is still subject to social marginalization, legal discrimination, and even physical violence, especially outside of the United States. How can we be trusted as members of society if Christians, who are the majority, expect us to rape and murder whenever we can get away with it?

Let’s indulge Robertson’s claims. While it’s easy to dismiss him as being on the fringe — and many do — atheists are often asked by religious believers what could possibly stop us from being violent. At a workshop that I once facilitated, I asked a room packed with atheist students how many of them have ever heard this question from a religious person. Almost every hand in the room went up.

Clearly, whether through willful ignorance or in-group isolation, many people don’t understand what secular morality is. Dan Linford, an adjunct professor of philosophy, does not believe that morality requires religion. He says, “Most ethical theories are (a) objective and (b) do not involve God.” He explains that some things, like being healthy and having control over your body, are intrinsically good. Other things, such as pain, are intrinsically bad. Forcing people to experience bad things and denying them good things is wrong, and rape denies autonomy and causes suffering.

Sarah Jones, a writer and church/state separation activist, draws her morality directly from her secular beliefs. “I believe that rape is a significant moral evil because I’m a secular humanist, not in spite of it,” she says, adding that her humanist views emphasize respecting the dignity of others. Blogger Niki M. says, “The concept of bodily autonomy doesn’t require religion. That is why rape (and murder and kidnapping) are wrong all the livelong day.”

Further, according to author and blogger Greta Christina, Robertson’s comments deny the fact that it is in our nature to try to treat each other well. “Compassion and a sense of justice are a fundamental part of what makes us human, part of how we evolved as a social species,” she says. “To deny our ethics is to treat us as less than human.”

Most people feel a sense of empathy, which includes not only being happy when others are happy, but also feeling pain when they are in pain. In fact, research shows that watching someone experience something activates some of the same brain cells as experiencing that thing ourselves. These mirror neurons, as they are known, may be the source of our capacity for empathy.

Maybe this can help confused people like Robertson understand how it is that atheists are no more violent than anyone else. “It has never occurred to me to go out into the world and rape or murder people because I’m an atheist,” says Kelley Freeman, Communications Associate for the Secular Student Alliance. “I don’t want to murder people because I am capable of basic human empathy.”

Some atheists would like to flip the question and ask Robertson how he knows that rape is wrong. “It’s interesting that someone would use the Bible as a source of morality, especially when it comes to treatment of women,” says Amy Monsky, Executive Director of the Atheist Alliance of America. “This is a book that says victims of rape should be stoned to death, after all.”

Yet it is atheists who are accused of condoning rape and expected to defend their morality–a hypocrisy that is keenly felt by those who have survived sexual assault. “I’m curious what Mr. Robertson would say to the man who sexually assaulted me,” says Courtney Caldwell, a blogger for Skepchick. “My assailant was a good ol’ boy, a Christian, an avid hunter. He was someone who would probably really enjoy Mr. Robertson’s show.”

Sarah Jones adds that, as a survivor of sexual violence, she knows quite well that rape is wrong. “I lived it,” she says. “I don’t need a sermon to tell me how to think about it.”

Perhaps some religious believers, who have grown up learning about the Ten Commandments and other faith-based morals, haven’t given much thought to where morality comes from if not god. It can be difficult to relate to and trust people when you have no idea what motivates their behavior, and it doesn’t help that vocal atheists are fairly rare when it comes to positions of power in America.

But atheists and believers aren’t as different as we think. Most people feel at least a little bad when they hurt someone, and most people feel good when they help others or give back to their communities. I trust people when I can feel pretty confident that they’ll avoid hurting me if they can.

That’s why statements like Robertson’s worry me and many others. Religious belief can wane or even disappear; many atheists were once believers. If belief in god is all that’s stopping people like Robertson from heinous acts of violence, that’s concerning, to say the least. “This is the kind of speech you would expect from a serial killer, not an educator or TV role model to tens of millions of impressionable Americans,” says Danielle Muscato, Communications Manager for American Atheists.

On the other hand, I doubt that Robertson and all the other religious believers who echo his sentiments would actually commit violence if they lost their faith. More likely, they are convinced that their religion is what keeps them acting morally because that’s what they were taught. People who conflate religion and morality may feel pain and guilt when they do something unethical, but may attribute those feelings purely to their religious beliefs and not to the fact that they are human.

But if ignorant people like Robertson really do believe that atheists are all potential rapists and murderers, so what? Unfortunately, that has consequences. Melanie Elyse Brewster, Assistant Professor of Psychology and Education at Columbia University, believes that opinions like Robertson’s have measurable, harmful impact on atheists in the United States. Citing research that suggests that atheists are highly mistrusted in our society, she says, “How are we supposed to be accepted in our communities, hold positions of power, raise families, collaborate with coworkers when some of them genuinely believe that we would rape or murder if it benefited us?”

I am an atheist whose stomach turned when I read Robertson’s speech. The atheists profiled here, who are all activists and leaders in the secular community, feel the same way. The only reason we are still constantly asked to prove that we oppose senseless violence — and the only reason I am writing this now — is because Christian worldviews are privileged in our society, and because the few atheist voices that get heard, such as that of Richard Dawkins, speak more about why religion is wrong than about why secular ethics are right.

Without a god telling us what to do, what’s left? Grappling with ethical questions on our own. But between the rich tradition of secular philosophy and our own neurobiological capacity to feel pain at the pain of another living creature, atheists have plenty of solid reasons not to commit violence. What we lack is the trust and respect we deserve as members of a society still dominated by Christianity.