As sexual assault and harassment allegations against powerful men from Harvey Weinstein to Kevin Spacey to Matt Lauer continue to pile up, it has become harder to justify this cultural moment as a simple case of separating the “good” guys from the “bad.“

Meanwhile, some men, according to a recent story in the New York Times, are confused about what sexual harassment even is, and whether they’ve done it. Jonathan Segal, a lawyer from the Equal Employment Opportunity Commission’s harassment task force told the Times about some of the “odd” questions he’s received from men since the Weinstein allegations:

Mr. Segal, who runs anti-harassment training, is now expanding part of the program called Safe Mentoring, which teaches men how to mentor younger women without harassing them. At a recent session, a male supervisor talked about having an extra ticket to a sporting event and feeling he could invite only a male colleague; Mr. Segal went over how to invite a female colleague without sexually harassing her. “The answer to harassment cannot be avoiding women,” he said.

Of course not, and men who want to enact the “Pence rule” (avoiding socializing alone with any woman who is not one’s wife) do so at a potentially enormous cost to their female colleagues, their organizations, and themselves. In fact, the notion that some men are confused as to how to “mentor young women without harassing them” is a troubling comment on masculinity.

And I mean masculinity, not men. Masculinity, after all, isn’t innate. Just as many historians and critics view race as an invention of racism, masculinity is a deeply entrenched cultural role that many leading thinkers and researchers believe has much more to do with social mores than biology—nurture, that is, over nature.

That’s not to minimize it. Indeed, many people misunderstand the incredible power of “nurture.” As a trans man, I can attest that injecting testosterone certainly changed my physiology, but my public appearance—which dictates everything about my place in the world, from my professional life to my relationships—was formed in reaction and adaption to the expectations of my environment, not the other way around.

Testosterone did not teach me to “walk like a man”—men did. Being an adult in transition has afforded me a perspective on male socialization that teenagers don’t get, and I’ve written extensively about how many of the rewards and expectations of masculinity should trouble people of all genders.

To understand how so many men, especially publicly progressive ones, could behave with such brazen inhumanity toward women—and how so many others could be baffled as to whether they somehow sexually harassed colleagues without even being aware of it—we have to look at how deeply men have internalized what sociologists call “toxic masculinity.” That’s the learned behavior associated with masculinity that endangers men and the people around them, such as excessive risk-taking, sexual dominance, and not asking for help.

These unhealthy cultural notions of “what makes a man” are increasingly seen as the root of domestic violence and mass shootings, as well as shorter lifespans and poorer health outcomes for men. As tempting as it is to put as much distance as possible between ourselves (or the men in our lives) and the “bad men” who are predatory and violent, the guys following the “Pence rule” should give us pause. The implication—that men “can’t help” ourselves or, worse, that we must protect ourselves from vengefully lying women–shows the incredible power of masculinity as a narrative.

Men start learning this narrative as boys—what being a man is, and what it isn’t. And the loss of intimacy, with other boys and men as well as with women, is central to it. Niobe Way, a professor of developmental psychology at New York University has spent decades conducting research with adolescent boys who, she says, are initially no different than girls their age when it comes to forming deeply intimate friendships. In her book, Deep Secrets: Boys’ Friendships and the Crisis of Connection, 15-year-old Justin says of his best friend:

You have this thing that is deep, so deep, it’s within you, you can’t explain it… I guess in life, sometimes two people can really, really, understand each other and really have a trust, respect and love for each other. It just happens, it’s human nature.

This language around friendship continues until they are about 15 or 16, Way says, when social concerns about being perceived as “girly or gay” overpower their intimacy needs. Boys (who are groomed unconsciously and very early to take risks and are generally given less of an emotional vocabulary by their parents) learn that the last stage before manhood involves rejecting connection with other men (“gay”), and vulnerability (“girly”). This is also, she notes, the time when the male suicide rates quadruples that of girls (adult men are also more likely to kill themselves). Social isolation is a key risk factor in poor health and violence, and it’s also a rite of passage for most American boys.

Today, it’s easy to identify the most toxic results of hyper-masculinity in the actions of sexual predators, who are finally being brought to justice. There’s some satisfaction in seeing them fall, one by one. But I remain worried about the men who live in shades of grey, men who have reacted to the ruin of offenders with wide-eyed defensiveness, tone-deafness, or palpable anxiety.

Can these men truly not know if they themselves have abused or harassed another person? It should not be difficult to tell when a person is uncomfortable with our behavior. After all, we are social animals, highly attuned to the cues of other humans. I suspect many of the men facing shadowy worries about past transgressions know that they did, in fact, behave badly.

But the lack of emotional intelligence that allows a man to convince himself that he is unsure if he has “crossed a line” at work is truly staggering—and worrying. These are the men who think they should follow the “Pence rule,” who are hand-wringing over the fear that they have “overstepped” in the past. They are likely working alongside you. They might even be living in your house.

Let’s be clear: This attitude is dangerous. It all-but-guarantees that this behavior will continue to be part of our culture, and the inheritance of our sons.

But the good news is that we can, and should, reverse course. We can create a culture where boys are not raised to see masculinity and humanity as mutually exclusive. But first we have to show them, and maybe even ourselves, that they aren’t.