I often walk around the city wearing a beard and a skirt. This is when I’m most myself, but it’s also when I’m most afraid of people’s reactions. As a South Asian nonbinary person – someone who does not identify as a man or a woman – I have grown accustomed to people being disgusted by me, to strangers calling out on the street, “What the hell is that?”

Media outlets – like Time magazine on its 2014 cover featuring Laverne Cox –proclaim that we’re at a “transgender tipping point”, a time of unprecedented visibility for trans people. Yet at moments when I’m facing aggression or contempt from strangers, I recognize that putting trans celebrities on pedestals doesn’t translate into safety for those of us who are visibly gender nonconforming.

Our culture still holds an ingrained suspicion of gender nonconformity, as if people like me exist solely to deceive and harm others. I remember all the times I have been called a freak, an “it” and ugly. To refuse to participate in the gender binary –the idea that there are only “masculinity” and “femininity” which exist in opposition – is to be considered a monster.

I do recognize that there have been some promising changes. The number of Americans who report knowing a trans person has doubled in the past seven years. Obama has hired his first trans staffer; major Democratic leaders have voiced their support for transgender rights; a trans television series even won an Emmy. But prominent trans figures still tend to fall squarely on one side of the gender binary – that is, they transitioned from one side of it to the other. Think Caitlyn Jenner’s pinup look in Vanity Fair or Ian Harvie’s lumberjack aesthetic on Transparent.

But not all transgender people medically transition into an opposite gender, contrary to how the mainstream media is telling this story, nor do we want to. Many of us do not change our names or documents, do not undergo hormone therapy and do not seek to pass as cisgender – when someone’s self-identification aligns with the gender they were assigned at birth.

For me, the “trans tipping point” tends to be yet another form of exclusion because it recognizes only those trans people who make claims to “real” womanhood or manhood. Transgender people who present a fixed male or female identity are regarded as representative of all of us. And I wonder if their acceptance by society is less a reflection of progress than a question of palatability. Indeed, while celebrities like Jenner challenge the idea that gender is innate, ultimately they don’t challenge society’s mandate that we all must exist as either male or female.

The rest of us – whose identities are more fluid, more difficult for strangers to comprehend and relate to – may not be visible in media but are more noticeable on the streets. As it stands, according to a nationwide survey by the National Center for Transgender Equality, nonbinary people, especially those of us who are people of color, are more likely than binary trans people to attempt suicide, be harassed by the police, live in abject poverty and be sexually and physically assaulted. What has become evident is that so many of us who do not pass as male or female are still regarded as disposable by both cis and trans communities. Too often, efforts to gain acceptance and rights for trans men and trans women has meant ignoring those of us who are not as easily categorized.

Take, for example, the issue of trans-inclusive public restrooms. Recently politicians in Florida, Texas and other states have attempted to pass laws criminalizing trans people for using public restrooms. A trans-activist photo campaign called #WeJustNeedToPee responded to the proposed laws with photos of trans people looking out of place in restrooms of the genders to which they were assigned at birth: a trans woman with long ringlets and red lipstick in front of urinals, a trans man in a cowboy hat and beard looking stoic next to a woman at a bathroom sink, with the caption “Do I look like I belong in women’s facilities?”

The campaign successfully highlighted the ridiculousness of the “bathroom bills”, but it did so by leaning on old-fashioned gender rules: shock that someone who looked like a “woman” could be in a “men’s” restroom and vice versa.

People like me were erased from this framing, even though we often experience the brunt of gender policing, because society continually misgenders us. Rather than challenging the idea that you can tell someone’s gender from what they look like (or the notion that bathrooms should be gendered to begin with!), many trans activists and allies accepted the idea that certain people who look certain ways belong in certain bathrooms. Nobody should have to look a particular way to pee safely.

I wonder if we can understand this “tipping point” less as a moment of triumph and more as a call for reflection. Society’s message to trans people feels like: “Congratulations! As long as you look like a conventionally attractive, respectable, thin cisgender model. Otherwise expect to continue experiencing discrimination, hostility and violence – and to be blamed for it.”

This is not the fault of individual binary trans people; it is part of a larger system of gender binarism that requires us to assimilate into discrete categories of “man” or “woman” to be worthy of recognition and safety. Instead of requiring trans people to modify how we look, our society can work to redefine its limited ideas of what masculinity and femininity are. Let’s push beyond the “tipping point” and ensure justice for the full range of our identities.