The Shit Girls Say meme, where men in drag make videos saying, well, shit girls would say, could be taken as an incisive summary of how the political is the personal; how something as arbitrary as gender determines our most basic lexicon. Or maybe it's just a joke.

I didn't get the joke: gender-based stereotypes don't make me laugh. I stayed thin-lipped when Shit Girls Say got racialised, and when Shit Black Girls Say, Shit Asian Girls Say and Shit Latina Girls Say hit YouTube.

But things took a turn when Franchesca Ramsey released Shit White Girls Say … to Black Girls, which quickly inspired Nicola Foti's Shit Girls Say to Gay Guys, and Sameer Asad Gardezi and Kosha Patel then unleashed Shit White Girls Say … to Brown Girls". Each video showcases a bewigged Ramsey, Foti and Patel reeling off a list of the most awful things your best white girlfriend has ever said. These videos skewer that verbal equivalent of friendly fire: friendly prejudice, if you will.

What's friendly prejudice? The most common defence of racism is: "But I didn't intend to be racist." This response relies on the idea that if we didn't intend to offend someone, then their feelings can't possibly be hurt. The Shit X Says to Y videos are delightfully validating because they show that those with the genuinely lovely intentions of being your friend and seeking commonality with you can still be rude and hurtful.

Unsurprisingly, the Shit X Says to Y meme has itself been called offensive. As a commenter on the NPR blog says, "if the roles were reversed … Jesse [Jackson] & [Al] Sharpton, would be involved, lawsuits filed, perhaps riots …" But the roles can't be reversed. The reason why relationships between white and non-white people, or straight people and gay people are fraught, is because of our history – long gone, recent or ongoing. Racist, homophobic or simply thoughtless comments are insulting not just in and of themselves, but because they are a bilious reminder of the times when straight, white people have dehumanised and denied other groups their human rights. Of course, non-white and gay people can say nasty or even prejudicial things to white and straight people, but those things don't deliver the sting that comes from decades of being on the wrong end of an unequal relationship (and could I recommend further viewing on this topic: comedian Kumail Nanjiani's "Racists").

What bothers some viewers about the Shit X Says to Y meme is that it targets only white women. Critics have said of Foti in particular that it is always sexist when men use women as the brunt of any joke. But privilege does not work in debits and credits, whereby your lack of cultural power as a gay person is paid back by your stores of cultural power as a man. A white woman can be racist to an Asian man, just as a straight black woman can be homophobic to a gay white man. These videos are important because they ask all viewers – regardless of what power they have and what power they lack – to reconsider if their best friendship with non-white and gay people grants them licence to cross the line.

Yet this doesn't change the fact that though white men started Shit Girls Say – to poke fun at white women – the backlash is against white women. Shit X Says to Y generalises these women as the only ones who possess a desire for intimacy or approval; that desire which bulldozes over the very real fact that, when differences in identity are at hand, there are parts of our friends' lives that we can never understand. Love doesn't conquer all. Still, it's unfair to put this burden squarely on them. The lack of a Shit White Men Say to Y meme (or Simply Shit White People Say to Y meme) is uncomfortable proof that we always prefer lampooning women than men.

And to that last group of the disgruntled, who object to the memes because they have plenty of gay friends or non-white friends (or non-white gay friends) to whom they've never said any of these things? Well, congratulate yourself for your thoughtfulness. And then do this: next time you see that black/brown/gay … person that you hold so dear in the line of friendly prejudice, step up. Suggest that those friendly words aren't so friendly after all.

• Follow Comment is free on Twitter @commentisfree