Paul Elam, the founder of men’s rights website A Voice For Men, has been railing against women, and against preferential treatment for women in custody battles, for decades. Which is funny, because it turns out he voluntarily signed away his parental rights, according to a BuzzFeed feature on Elam, complete with an interview with Elam’s first wife and daughter. Also, he’s an advocate for young boys who are abused, but he’s cool with spanking his grandson for opening a refrigerator, and although he has published some really nasty stuff about women, he’s been totally fine with living off of his girlfriend’s money, according to his daughter.

So there’s that. But honestly, I’m kind of shrugging at the news. Paul Elam is exactly as much of a hypocrite as most people, including the people who read A Voice For Men. He’s just a really loud hypocrite. I’m not as concerned about his hypocrisy as I am about the extremity of his rhetoric, and how much we’re all leaning on extreme rhetoric when we talk about gender.

I care in general about as much about Paul Elam as I do about trans-exclusive radical feminists or sex worker-exclusive radical feminists or women who truly and genuinely hate men. Paul Elam, and men like him, are as much in the minority of men as TERFs and SWERFs, and as real, actual misandrists are in the minority of women. Ultimately, those two groups—extreme men’s rights activists and extreme feminists—are only fighting with each other, and they’re just doing it very loudly. Everyone else is caught in the crossfire.

Everyone else is caught in the crossfire.

What that crossfire looks like is this: In October, I saw a really upsetting tweet from a troll about how women should expect more murder and violence over sexual or romantic rejection. I said, in front of my boyfriend, “Fuck men.”

That spurred a long, heated argument in which neither of us was just saying that we were hurt. My boyfriend felt like I was being extreme and defending a misandrist ideology and lumping all men in together. I felt like he was being extreme and defending a misogynist ideology and ignoring the suffering that so many domestic violence and rape victims go through, and ignoring how frightening it is to be made a constant sexual object in public. I didn’t see that he wasn’t trying to defend a misandrist, but just had real, genuine insecurities about how I conceptualize average men as a group, and him as a part of it, because of a statement I made in the heat of feeling frightened by a different man’s hateful ideology. He didn’t see that, despite the fact that I’ve done a lot to heal from my traumas, the terror I feel about violent men is still very, very real and present for me every day.

We were both just really insecure about how we should expect to be treated or perceived by people of other genders. Should he expect to be hated because he’s a man, regardless of whether he ever did anything wrong? Should I expect to be violated because I exist as a woman, and some people see that as reason enough to hurt me?

Men’s rights activists talk about their “red pill” moments, the moments when they realized that there are gender double-standards that are detrimental to men, that men aren’t always treated fairly, despite progressive political language about systemic bias that says that men are by and large the beneficiaries of our economic and political systems. (That is demonstrably true: Look at the makeup of the American Congress—it’s 80 percent male. Look at the Fortune 500 list: Only 24 of the 500 last year were women.)

But I don’t doubt men who say that their mothers beat them with impunity, or that they’re bothered by women who are habitually violent toward men with the expectation that those men shouldn’t self-defend, that they feel like their only social worth is in their earning potential, or that they’re frustrated by being told that they should be aggressively sexual in order to be manly but that they’re also then creeps for being aggressively sexual, and so on. I don’t doubt that men don’t feel heard about the things that hurt them, and I don’t doubt that it’s frustrating when a woman who is in the same exact situation—feeling hurt, feeling unheard—tells them that their pain doesn’t matter because of the patriarchy or whatever.

The terror I feel about violent men is still very, very real and present for me every day.

But there it is. Women are in the same boat. We have “red pill” moments, too—when we’re condescended to by our male peers, when we’re passed over for promotions in favor of less-qualified men, when we’re raped, when we’re beaten, when we’re molested, when we’re manipulated and abused, when we find out behind closed doors that most of our female friends have also been raped or beaten or molested or manipulated or abused, when we feel like our only social worth is in our sexuality, and when we’re solicited for sex or sexual attention all the time, but then told that we’re sluts if we act on our sexuality. And then we have men telling us “not all men!” instead of giving us just basic sympathy when we recount our pain, or we’re told that we’re lying, that we just don’t experience that pain at all.

Men and women (such as we are) who care about gender issues either way really need to stop viewing this conversation as if we’re at odds with each other, as if there’s not enough freedom or compassion in the world to go around, as if legislation that benefits one group necessarily harms the other, or as if legislation that hurts one group protects the other. When we see each other as “at odds,” we tend to paint each other in broad strokes, when the reality is that every feminist and every men’s rights activist who really cares about the issue has had complicated personal experiences that have led to a certain level of distrust. That is the case as much for men as for women.

We see each other as if we’re on opposing sides of an issue.

We see each other as if we’re on opposing sides of an issue—so on sexual harassment, for example, women resent being treated as objects and men resent being pressured into having to initiate. But it’s not us individual women who have dictated that the rule is that men initiate, and it’s not individual men who have dictated that women are only interesting inasmuch as we’re sexual objects, even if some women really think that’s the rule and some men really don’t care about women except for our vaginas. Rather, it’s the big pile of assumptions about gender that our culture has been building on for the last several centuries.

We’re all born into this culture, and we’re told how to act, and we feel let down when our actions don’t elicit the response that we were told they would get, when we don’t benefit from playing by the rules that our culture sets for us.

Laying blame on other people feels easy, because we assume that if we can persuade or bully those other people into acting in a way that’s more in line with what we expected, we can solve the problem. Instead, of course, it’s just wound up making the problems worse. A man gets sexually rejected after doing what they were told to do and initiating with women, and it’s easy to blame the woman for rejecting him, or blame feminism for saying that his behavior is creepy. A woman dresses conservatively because she’s told that will protect her from men’s advances, but then gets catcalled every time she leaves her home anyway, and she blames men for overstepping the boundaries she was told they wouldn’t if she followed the rules.

What really crystallized the sort of subnarrative of my fight back in October was when I asked my boyfriend if he believed the things the troll did. He asked, “Don’t you trust me enough to know that I don’t?” And in response, I asked, “Don’t you trust me enough to know that I don’t hate men?” There was so much we weren’t saying to each other, and so much we were assuming without clarifying with each other.

Likewise, I’ve assumed for a long time that, for instance, men who catcall me are intentionally trying to make me feel threatened, regardless of their professions that they’re trying to pay me a compliment. I can’t even imagine the number of assumptions that are made about me based on the fact that I’m a feminist, and all the opportunities to explain my experiences, all the opportunities to sympathize with other people, that I’ve missed because those people are leery of feminism and have written me off from the get-go.

Laying blame on other people feels easy, but it’s just wound up making the problems worse.

We don’t need more Paul Elams, spouting off generalizations and advocating for violent language that makes other people feel more defensive. We need sane conversations about our expectations of and assumptions about men and women that take place in calm, controlled, private settings. We need more respect for each other’s humanity. I can extend that to someone like Paul Elam by saying that I don’t care as much about either his rhetoric or his hypocrisy as I do about the fact that he’s a complicated human being who obviously wants his feelings to be heard, or else he wouldn’t be shouting about them so loudly and with such vitriol.

So there’s my first step. What’s yours?

This post originally appeared on the Frisky and has been reprinted with permission.

Read more from the Frisky:

12 Good Things about Being Stuck in a Rut

I Was Blessed By Levar Burton

Photo via MCAD Library/Flickr (CC BY 2.0)