If I have learned anything in my life so far, it’s that the only group of people more obsessed with touching a penis than gay men is straight ones. Promise.

I began noticing this all the way back at my very white elementary school, when boys would roughhouse and grope each other on the playground while always making sure to punctuate their grabs with gay slurs that called the receiver of that grab a homosexual.

As I got older, those grabs evolved. And over time – especially once I got to my very white college – the grabs from straight men became caressing or kissing or, for the bold, sex. And during all of this, these men, these straight men who were always my bully growing up or even in college classrooms, maintained their straightness while I was constantly reminded of how they despised my gayness even as I entertained their episodic gay-interests.

And I am not alone in being the object of ambivalent, conflicted desire by men who identify as straight.

“I think homosexual desire and homosexual contact are staples of the human experience,” professor Jane Ward, University of California, Riverside, recently told me. “But are also subject to incredible cultural baggage.”

But why are all these straight men vying to take a swim in the rainbow pool?

Ward, who recently published her latest book, Not Gay: Sex Between Straight White Men, asked herself a similar question years back after hearing from a man she went on a date with about how ‘gay’ his straight fraternity had been.

From her research, she has arrived at an interesting conclusion: straight men – specifically white men – are having sex with other men to affirm just how straight they are, because to be straight and still be able to perform ‘gay sex’ – while always remaining uninterested – is the height of white masculinity. And they are the primary group doing this, because they can.

“Sometimes white people, and men in particular, bristle at the concept of ‘privilege,’” she says, speaking more broadly about the term that many use to describe inherent advantages white people have due to skin color.

“But in the context of [my] book, recognizing privilege isn’t about denying what is unique about individual straight white men; it’s about recognizing that straight white men have some unique cultural resources they can draw on to explain away and justify their presumably discordant sex practices.”

According to Ward, this behavior is very much tied to their white privilege, heteronormativety and male privilege to create a nexus in which straight white men can have sex with one another and face no repercussions.

“White men have more room to push sexual boundaries without being immediately pathologized [due to their privileges],” she continued.

And she’s right – and I am annoyed that she is on a certain level.

So far this year, LGBTQ people of color – especially transgender women of color – have faced record-breaking amounts of violence – a 20-year-old transgender woman was murdered in Dallas this week. Her name was Shade Schuler.

The group that faces less violence, while perpetuating it the quickest in our current moment, is straight white men.

I don’t have an issue with straight men having sex with other men and not calling it gay or having it change their identity. People can and should do whatever they desire as long as it’s consensual. But what I find annoying is how this game called life is so unevenly stacked – with one group holding all the cards.

We see that with the disproportionate ways in which people of color face police violence, poverty, health disparities and this list could keep going.

As a gay man, who has faced violence for being gay, to see evidence that shows the very men that perpetuate this violence are doing the same sexual acts as me to show just how ‘straight’ they are is absolutely gross - and homophobic at best.

(Or how problematic is that white straight men get a whole book in many ways defending their straightness, but black men are most of the time demonized with words like ‘down low’ in books about their lives.)

Being gay is still not easy, especially as a person of color. And thanks to the help of the marriage equality movement, being gay is becoming less and less gay, and much more straight – with many seeing ‘us’ as finally close to being straight. This thought has even led to our gayborhoods beginning to disappear as acceptance of LGBT people rises.

In 2015, gay seems to be less gay than it ever has been. And while I want acceptance of us in the world at large, I still want us. I want us queer, I want us to have individualistic characteristics as a group, I want us to have something that is ours and not something that a straight white man can play with to prove just how much of a man he is.

But what I don’t want is to hear that white privilege not only lets straight men get paid more than me, face less violence than me, live longer than people like me, but also have sex with other men and not facing any of the violence people like me face – because that is incredibly infuriating.