Any woman can spot an anonymous misogynist. A man might flick his tongue as we pass on the sidewalk, or sprawl all over a seat on the bus when we’re carrying three bags. He might jerk off in front of us or be that online dating match who sends an unsolicited cock shot. A man we’ve barely met might interrupt us on a panel, or offer litanies of their accomplishments without bothering to hear about ours. A man we see on TV might punch his fiancée in an elevator, or prey on underage girls, or repeatedly get accused of rape.

We’ve come to a point in the gender wars where certain repugnant male behavior is no longer up for debate among reasonable people. Republicans have taken up the mantle of domestic violence; Megyn Kelly will grill Donald Trump about sexist tweets on national television; spousal rape has gone from acceptable to illegal in all 50 states. Even the lesser offenses are universal enough that we now have terms for them — some that seem old-fashioned, like “catcalling”; others that have recently sprung from the internet like “manspreading” or “mansplaining.”

And yet one of the major obstacles in the Fight Against Patriarchy is that a sexist guy will always seem like an outsider — a bad-news ex-boyfriend, perhaps, but not your male feminist friend, your super chill brother, your gentle dad. Never the bros you know and love, never the “fair-minded guys who want women to succeed.” Never one of Our Guys.

But realistically, mathematically, it doesn’t add up: We must know some of these dudes. It wasn’t always so easy for our supposed male allies to hide in plain sight; when radical feminists first burst on the scene at a New Left rally in Washington, DC, in 1969, progressive hippie men screamed at a women’s libber: “Take her off the stage and fuck her!” Paradoxically, now that such blatant sexist behaviors are in theory no longer tolerated, we convince ourselves that the specific men in our lives would never engage in them. It’s why the concept of date rape still provokes the deepest self-doubt in the most confident women: a playful peer, a friend, wouldn’t cross that line.

I felt a pang of this self-delusion the weekend Stoya tweeted that James Deen had raped her. I never quite drank the James-Deen-is-a-feminist Kool-Aid, but it had been comforting that he was a boy I recognized, someone who could’ve gone to my summer camp or Hebrew school. He, as Kitty Stryker wrote on Medium, “was supposed to be one of the good guys.” Shamefully, my first reaction to the tweets was to hope Stoya was lying. Because if James Deen is a rapist, what does that say about the guys he reminds us of?

When I witness sexists in action, I almost don’t think of them as humans. But that guy who demonstratively ogled me on the street the other day, the guy whose face I can’t even remember? He has friends and family living in a parallel universe where he’s a decent person. Thelma and Louise asked the truck driver pointing to his lap and waggling his tongue: “How’d you feel if someone did that to your sister, or your mother, or your wife?” Yet we never quite ask: “What if your sister, or your mother, or your wife knew you were a harassing piece of shit?”

This cognitive dissonance can be distilled into that little phrase, brilliant for its simplicity: Not All Men. The meme dates back to 1985 when Joanna Russ, in her feminist novel On Strike Against God, wryly repeated the bad-faith motto. It’s the echo of every man wishing to exempt himself from misogyny by pointing out that not all men — and especially not him — manspread, mansplain, manterrupt, catcall, harass, intimidate, hit, rape. But it’s also every woman hoping he’s right, hoping that the world she’s created doesn’t include such faceless creeps. Even Stoya said it took “months and months” to call her rape what it was. The internet fuses these two opposing forces in ugly ways, like when you see a guy friend liking pics of scantily clad teenage girls on Instagram, or when a man’s unhinged outrage over a polite romantic rejection occurs against the backdrop of family-friendly Facebook. But even those wake-up calls aren’t enough to convince us that those two worlds are, in fact, just part of one.

When we see or hear that our dudes have done some of these things, we make hideous excuses for them. In 1989, the idyllic town of Glen Ridge, New Jersey, protected its star athletes after they gang-raped a mentally handicapped girl in a basement. Similarly, in 2012, when several football players in Steubenville, Ohio were accused of sexual assault, a social media trail burnished behind them, much of the town fiercely defended their boys; the grandmother of one of the accused said he was just “at the wrong place at the wrong time.” Last month, news broke of an entire Missouri town that couldn’t fathom that their “good man” had sexually abused a girl more than 300 times over 13 years — even when he confessed. “Darren is one of the most admirable people I know,” wrote a friend of the convicted rapist, in a letter begging the court to go easy on him.

Well, that’s Middle America, the cosmopolitan among us might think. But I’ve made these excuses, too. In 2004, I was getting sloshed at a bar with a bawdy, outrageous girl I had just met. Eventually I found out she went to the same college as “Chris,” a guy I’ve known since forever. When I cheerily asked whether she knew him, she replied, fiercely, “He raped my friend.” She told me “everyone knows to stay away from him.” She told me he was a “shitty person.”

It was a needle-off-the-record moment, the impact of which has obscured my memory of exactly how I replied. I must have acted defensively; I know I didn’t ask for any details. I still see Chris—whom I remember as a kind-hearted kid with a beatific and liberal mother—once or twice a year, at which point I desperately try and fail to erase that decade-old memory. His every remark about girls or sex or parties seem loaded with meaning. The last time Chris and I chilled, he was drunker than I’d ever seen him. When he put a hand on my back for a split second, it gave me shivers.

I’ve engaged in endless rationalizing, denial, excuses that teeter dangerously on the side of victim-blaming: “Maybe she’s exaggerating,” or “Maybe they were both too drunk to remember,” or “Maybe that chick I met at the bar is just crazy, or mistaken, or a pathological liar.” That “gray rape” concept I once rolled my eyes at? This must be the one true case of it. Still, even now, I refuse to believe it’s true. I would rather die than ask him about it.

We’re all gripped by a constant negotiation and cycle of forgiveness with the people in our lives. A gay man will wave off a whiff of homophobia from his sister because he knows “she’s not like that.” A woman of color will ignore racially off-kilter comments from her white friends because she knows where their hearts are. Any of us who’s stayed close with less-than-perfect people has needed to reach for some form of denial and focused on the reasons why we love them. Forgiveness is a virtue, right?

It’s painful to think about what we’re burying, but the alternative is equally undesirable. It would mean seeing every human as an insignificant speck of a larger problem rather than balls of contradiction and messiness. It would mean not trusting anyone. And it would require giving up on a staggering number of people, especially Our Dudes.

But maybe we don’t have to choose. Maybe we can confront the misogynists we love head-on, precisely because we love them. We owe it to the men in our lives to call them out; we owe it to other women who shatter our compartmentalized views of guys we think we know. At the very least, we owe it to ourselves to admit that no man is immune to patriarchy. In his devastating profile of R. Kelly, wherein he asks how we can bump to the grooves of an alleged serial statutory rapist, David Marchese employs a quote from F. Scott Fitzgerald: “The test of a first-rate intelligence is the ability to hold two opposed ideas in mind at once.” The most evolved state of all is loving our dudes and hating their faults. We just have to stay woke while we do it.