We need to talk. About bathrooms.

Honestly, I don’t remember a time when so many angry words, backroom political maneuvers, votes and vetoes, protests, counter-protests and economic boycotts were prompted by the question of who gets to use which toilet.

And yet here we are: Whether transgender people should be required to use the bathroom labeled with their gender at birth, or the one labeled with the gender they identify with now, is the subject of passionate debate and legislative action in North Carolina, Mississippi and other parts of our country. Why?

Bathrooms are the intimate, yet public, stage on which our anxieties about shifting sexual norms and identities are being played out. So while on a practical level this debate is about the rights (or restrictions) of transgender people, if you listen closely, it’s about so much more.

“We are in culturally uncharted waters,” explained Joy Ladin, the chair of the English department of Yeshiva University’s Stern College for Women, and the first openly transgender professor at an Orthodox Jewish institution. “We need to figure out what it means to live in a society that acknowledges the existence of trans people.”

And we have to be willing to acknowledge that this is not an easy task, even for the willing.

There is no denying that on a basic level, these debates are driven in part by homophobic bigots and their funders, who want to scuttle laws that don’t do anything more than protect trans people who have to pee or shower or change their clothes. The supposed threats that these protections pose to others are largely nonexistent; in fact, it is transgender people who are far more likely to be the target of violence than any random user of the ladies’ room.

Keep in mind here that the U.S. Department of Justice estimates that one of every two transgender individuals is sexually assaulted or abused at some point in his or her life.

But in our heightened public discourse, actual facts seem to matter little. Joel Hunter, senior pastor of Northland: A Church Distributed, a mega-church that worships at three sites in Central Florida and reaches many more online, told me that even though there are transgender members of his congregation, the evangelical Christians he knows are still fearful of the bathroom mandate — and, more broadly, fearful of a hyper-sexualized society where boundaries are eliminated and gender confusion reigns.

“This is unfamiliar territory,” he said, speaking by phone from his Orlando church. “There is a general apprehension in the evangelical community about the rate and speed of change that the culture is undergoing right now.”

I heard other justifications. Matt Sharp, legal counsel for Alliance for Defending Freedom, said his organization opposed the transgender bathroom mandate because it would violate privacy and the value of modesty that, for him, is based on Scripture.

Still another evangelical leader I spoke with, who asked not to be identified, had more practical concerns. How does he accommodate transitioning students in his church-run school? How does he ensure modesty? Calm anxious parents? Afford to put cameras in the bathrooms so that all students behave? Build stalls that lock and sinks that don’t touch so that all students feel safe?

Never mind dealing with members of his flock who believe that changing one’s gender is a sin.

“I want to obey the government. How do I do that?” he asked me, and I don’t think it was a rhetorical question.

And then, when you really consider the fears expressed by those opposed to these changes — real and, too often, imagined — you realize that these people aren’t freaking out only about trans people. They are freaking out about men.

The bathroom — more specifically, the women’s restroom — is the one publicly sanctioned refuge for women, a sanctuary from harassment and abuse, a safe space, to employ that overused phrase. In all my conversations and in all the articles I’ve read on this issue, I haven’t heard anyone express the fear that a person who was born a woman and now identifies as a man will cause harm if he enters a men’s room.

But the fear that a person who was born a man and now presents as a woman will lurk behind the stalls to attack unsuspecting daughters is threaded throughout objections to the bathroom mandate.

“We’re in denial about most of the violence against women in our society, and this brings it to the surface,” Ladin observed. “If I am in a women’s room and see someone who looks like a guy, I’m taken aback. It doesn’t make sense, except for the fact that women everywhere are exposed to violence.”

In our conversation, Sharp emphasized what he contends is the growing risk that women who are sexual assault victims will be traumatized all over again if they are exposed to male genitalia in a restroom or locker. A small group of such women are lobbying to reverse the new bathroom policy in Washington state, making just this argument.

To me, this scenario is so extremely unlikely that it should hardly shape public policy. But it does illuminate the larger issue: This isn’t about transgender individuals behaving badly, it’s about men behaving badly.

So let us not deny the safety of transgender people because we are in more pervasive denial about the violence against all women that persists — indeed, seems to be expected — in our society. Civil rights legislation like this is a necessary first step in becoming a more inclusive and tolerant nation, but we need deeper conversations about what really underlies our fears and about the genuine threats to our safety. Like I said, we have to talk.