Some years ago, I worked for a man several decades older than me who consistently made oleaginous and sexualised comments to me and the other young women in our workplace. He was completely oblivious to the fact that our disgust towards him was a shared point of bonding, and that we would wince every time he was in the building; I expect because every time he said something we would force a smile, entrenching his delusion that we actually enjoyed his behaviour.

I thought about him again this week when I read that the Manchester teenager Gabrielle Walsh had been knocked unconscious after she told a man who followed her from a nightclub: “I’m sorry, I’m not interested.” Although Walsh’s experience is much more extreme and frightening than mine, both examples reveal the reluctance women feel to rebut a man’s unwanted advances when he holds some sort of power – be it physical or economic. In the wee small hours when alcohol is flowing, perhaps a man will just be crazy enough to physically harm you if you tell him no. If that man is your boss, maybe you’ll find your work life becoming that little bit harder after you inform him that what he regards as swashbuckling charm is actually sexual harassment.

But men don’t have to hold direct power to make rebutting their advances an unnerving experience for women. The philosopher Sandra Lee Bartky compared patriarchy to the panopticon, a type of prison that allows a single security guard to see every inmate while the inmates can never tell whether the guard is looking at them. Feeling as if they are constantly being watched, the inmates are motivated to behave themselves at all times, just in case. Similarly, argued Bartky, most women have a “male connoisseur” in their consciousness, and they are perpetually aware of themselves as existing under the judgment of a male-dominated society, even when men aren’t actually doing anything to reinforce this understanding.

In other words, men don’t need to directly threaten women for it to be scary to say no to them. We know that we are expected to receive their advances demurely, pliantly and with gratitude, and simply the awareness of that can be enough to keep us in line. Sure, maybe the guy won’t hit you. But maybe he’ll call you frigid, maybe he’ll accuse you of leading him on, maybe it was actually your fault – were you too friendly? Did you laugh too hard at his jokes? Did you stay talking to him too long? Sometimes the unease that comes with turning a man down isn’t even this fully formed: it just feels transgressive in some unspecified way.

Walsh herself alluded to the unspoken pressure women feel when she told the Manchester Evening News: “Girls feel like they can’t say ‘no’. They feel like if they say ‘no’ then [men] might hurt you and in this case it was true.” This is the side of sexism that we don’t talk about often enough: that the expectations placed on women are so pervasive that men often don’t have to do anything at all to keep them up. Yes, sometimes sexism is as conspicuous as a man punching you in the face because you turned him down. But other times it’s a thing that resides in your brain and regulates your behaviour. It’s the thing that makes you feel as if you can’t say no before any threat has even made itself apparent.

Facebook Twitter Pinterest ‘When I spoke to female friends during #MeToo about their experiences of sexual harassment, what struck me the most was not how traumatised they were but how casual.’ Photograph: Alain Robert/SIPA/REX/Shutterstock

When I spoke to female friends when the #MeToo movement was at its height about their experiences of sexual harassment, what struck me the most was not how traumatised they were but how casual. That’s not because sexual harassment isn’t a big deal; it’s because many women have had to alter their psychic geography in order to exist in a world where these forms of sexism are so commonplace. We have learned how to dull the part of us that feels distress when we are violated, because it’s impractical not to. Far too many women have accepted this state of affairs as just part and parcel of inhabiting their gender. Perhaps we don’t talk about this reality as much because it is so much worse than the idea that sexism is a matter of moustache-twirling villains preying on fragile women.

There's no duty on employers to prevent sexual harassment. That has to change | Emily Reynolds Read more

I believe we can organise society differently. As well as obvious protections to prevent women from being harassed at work, and harsher penalties for gendered violence, we can organise around a better set of values. We can raise boys with a stronger sense of empathy, respect and humility. We can represent women better in public life, and tell their stories. We can acknowledge that Gabrielle Walsh was not punched in some freak incident by a person who was born evil: she was attacked by a man who felt entitled to her because he lives in a society that has encouraged his entitlement. We can make sure the girls growing up today experience something better, but doing so requires the courage to change the society we live in now.

• Ellie Mae O’Hagan writes about politics and culture for the Guardian