I’ve heard a bit, this week, about “The Gay Agenda” in the wake of a ruling holding that a professional wedding photographer can’t refuse to shoot a same-sex marriage. I’ve also seen considerable irritation sent in Steve Horwitz’s direction after he (again) reminded libertarians that people do judge us on the company we keep: in this case, Ron Paul attending a conference populated by people who deserve the epithet “wingnut.”

The two issues are, I’m afraid, related.

Libertarians are rightly angered when representatives of disadvantaged groups run to the state in order to get their discrimination problems solved: this is the basic reason so many dislike feminism, affirmative action, and gay rights. Libertarians often argue that the biggest oppressors of those groups have historically been states.

And it’s at that point that the analogy breaks down.

If you are black, then yes, your worst enemy has always and everywhere been the state. A state was required to maintain and enforce the legal framework for slavery, Jim Crow, and many other exclusions, in the US, the UK, and elsewhere. To that extent, the libertarian argument is correct. However, if you are female or LGBT, then, historically, your biggest oppressor was a non-state actor–a monotheistic religion, either Christianity or Islam. To the extent that the state has oppressed women and particularly gays, it has done so because a religious group captured its levers of power and then used them against unpopular groups.

One only has to consider the different treatment of gays in Europe’s pre-Christian civilizations (Greece and Rome) to understand this. If you’re LGBT, it’s possible to imagine a state that isn’t your enemy at all; some pagan civilizations stuck LGBT people on a bit of a pedestal. If you’re a woman, it’s also possible to look at pagan Roman law and see a framework of rights that wasn’t matched anywhere until the 20th century.

There’s a reason the SCOTUS rejected an attempt by a Christian body to build racism into its doctrine. Doctrinally, both Christianity and Islam have to perform weird contortions in order to hate black people: that’s why there’s no evidence of racism in antiquity or the medieval world. Well, there is, but it’s directed against Jews. The first bit of “Jim Crow” this lawyer encountered was the Christian Roman Emperor Justinian’s prohibition on Jews and non-Jews marrying each other…in 529 AD. When it comes to LGBT people, however, hatred is doctrinal.

That’s why it matters when Ron Paul lends his name to Gary North’s homeschooling curriculum. That’s why it matters when libertarians don’t just defend scoundrels in the name of free speech but join them. And it’s why it also matters when libertarians support voluntary communities based on conservative religious principles.

Because LGBT people occur randomly in the population, it’s terrifying to contemplate the prospect of being born into one of those communities, surrounded by people who hate us. Worse, we have often called on the state to facilitate our ability to flee–what philosophers call “a right of exit.” And thanks to acrimonious debate over marriage equality, the relationship with many Christians is currently rocky. That’s why–in defending freedom of association–libertarians need to take great care to avoid endorsing the form of that association.

Because refusing to serve someone because they’re gay, or female, or black is pretty indefensible, really.