The Supreme Court wisely decided Monday not to hear the appeal of a New Mexico photography studio that didn’t want to take pictures at a gay wedding for religious reasons. Good call. The owners of the studio lost in state court because New Mexico—like some states but far from all—has a law protecting against discrimination on the basis of sexual orientation. By taking a pass, the Supreme Court seems to be saying: This is a state-by-state issue at the moment, and there was no need for us to interfere with the balance New Mexico has opted to strike between religious rights and the rights of gay people.

But claims in the name of religious liberty are very much alive in other cases, some involving gay customers, and also at the Supreme Court in the pending decision in the Hobby Lobby case. The craft store, as you may have heard, is challenging the requirement in Obamacare that employer-provided health insurance include coverage for certain forms of birth control. The link between fundamentalist objections to birth control and to gay weddings makes the meaning of religious liberty one of this year’s biggest legal battles.

I’ve tried to defend a version of religious liberty that balances religious rights against the costs they impose on other people. (My views are heavily informed by the work of a group of scholars including Brooklyn law professor Nelson Tebbe.) In a couple of thoughtful posts, New York Times columnist Ross Douthat responded that my defense of religious liberty is really no defense at all, arguing that it doesn’t concede “an inch to actual-existing dissenters.” I’m only in favor of pluralism for the “truly powerless,” he says—the Amish, as opposed to evangelicals. Other conservatives have also weighed in, and they don’t like my conception of religious liberty either. This is deep and important stuff. So here’s Round 2.

The Hobby Lobby case pits the religious freedom of a business that employs 13,000 people against the rights of its female workers to comprehensive preventive care. The idea that such care should include coverage for birth control comes first from Congress, which mandated preventive coverage when it passed the Affordable Care Act. And it comes from the Institute of Medicine, which has plenty of research to cite showing the health benefits of averting unintended pregnancy. The first question about this case is whether a business has religious rights—in lawyer’s terms, is it a “person” within the meaning of the Religious Freedom Restoration Act? I would say no, and more about why in a minute. But even if the Supreme Court mistakenly recognizes Hobby Lobby’s claim to religious freedom (the company is privately rather than publicly owned, which could make the difference), here’s the remaining key question, posed by Justice Anthony Kennedy at argument. As Kennedy pointed out, Hobby Lobby’s view that it need not cover birth control would allow “the employer to put the employee in a disadvantageous position. The employee may not agree with these religious beliefs of the employer.” Kennedy then asked: Do “the religious beliefs just trump?”

My answer is no. The Religious Freedom Restoration Act says the government can burden religious beliefs if it has a compelling interest in doing so and uses the least restrictive means. As I learned from Tebbe, in the past the Supreme Court has ruled that preventing a religious group from shifting the costs of accommodating its beliefs to another group or individual counts as compelling interest. The government shouldn’t exempt Hobby Lobby from the contraception mandate because that will force its workers to bear the cost of contraception.

As Linda Greenhouse pointed out, the Supreme Court could compromise by letting Hobby Lobby offer contraception coverage through third-party insurance carriers, which is what religious nonprofits organizations are allowed to do (though some of these groups object to signing the relevant form). But Douthat thinks my whole balancing approach is wrong. By siding against Hobby Lobby and the companies that have refused to bake cakes or take pictures or arrange flowers for gay weddings he says, I’m stifling dissent, in an overabundance of self-certainty:

Because as Bazelon’s blithe (and increasingly typical) dismissal of current religious-liberty concerns suggests, it’s precisely when people in liberal societies see themselves as out on the vanguard of history that they’re least likely to concede that they might, just might, be making a mistake, and most inclined to feel instead that the thing to do is shatter the shield wall around the remaining bastions of unenlightenment rather than permit them to persist. It’s when a consensus is at its most self-confident, in other words—and therefore most vulnerable to the errors of overconfidence—that the kind of pluralism that might serve as a corrective becomes hardest for that consensus’s exponents to accept.

This is where I’m supposed to show a little humility. Hmm. The thing is that I am certain of the benefits of both gay marriage and birth control, based on the evidence. Like I said, the Institute of Medicine knows what it’s talking about on the benefits of access to contraception. And while Hobby Lobby says it doesn’t want to cover the IUD and the morning-after pill in particular, because it sees them as abortifacients, that’s a seriously shaky claim. Even more so the claim that gay marriage hurts kids, or anyone else, which has been thoroughly debunked. Opposing these aspects of sexual modernity, as I put it in my last piece, is a pretty tough position to defend, rationally as opposed to theologically speaking.

My conservative critics will surely say I’ve revealed myself as irredeemably smug here, but honestly my conception of religious liberty doesn’t rest on who is right or who is wrong. If I thought the IUD caused abortions, I’d still argue for balancing religious freedom against other values rather than deploying it as a trump card. God bless our tradition of religious pluralism. But crucial to it, always, has been the understanding that the views of one religious group can’t trample all over the rights of another religious group or the rights of nonbelievers. That’s why the Constitution bars the establishment of one religion. We all get to live here.

Here’s another way to get to a similar pluralistic result. I am all for dissent and criticism by religious folk of all stripes. Shout it from the rooftop of your house, or your church, or your school or missionary group or your service project. Just don’t invoke the shield of religion if you’re the owner of a business who has chosen to incorporate for profit. Because at that point, you are shielded from personal liability. You’ve opted to become a “legally distinct entity,” as the Supreme Court has put it. For all intents and purposes, because it serves your interests, you’re not you anymore.

Unless I’m missing something, Douthat didn’t address this part of my argument. He said I’m only willing to allow for the religious liberty of powerless groups, like the Amish, who won the right to keep their children out of regular school, in the face of a state law that required attendance, back in 1972. Douthat points out that if the Amish kids disagreed with their parents, then the cost to them was real and searing. True. But courts generally treat the family unit as one in this kind of case, letting the parents speak for the children, and I was going along with that legal framework.

Here’s another Supreme Court decision I’m OK with: a 1987 ruling in which the court allowed the Mormon Church to fire a building engineer for “failing to observe the highest standards of Mormon belief and practice.” The engineer was building a gymnasium run by the church, which isn’t the same as building a temple. But the court recognized the trickiness for a church of separating its religious activities from its secular ones. And the justices emphasized the nonprofit nature of the church as a whole. So this ruling makes sense to me, even though it does impose an obvious cost on the guy who got fired. I can see balancing the scales differently when a church itself is involved. And I’d feel the same way if another case like this substituted Catholic or Baptist for Mormon.

At the National Review, Yuval Levin says my conception of religious liberty comes out of a progressive vision in which “society really only consists of individuals and the government, and that the various institutions and power centers that lie between the two are inherently illegitimate and should either be seized or pushed aside when they get in the way of public-policy objectives.” I hope I’ve made it clear why this is not an accurate description of my position. I’d group churches and nonprofit organizations—just not businesses—with individuals. The history I keep flashing back to is the sit-ins at lunch counters in the South in the 1960s. Those businesses invoked their right to refuse to serve black customers. The courts said no, because they were open to the public. If you are a public accommodation, you have to be open to everyone. The same has to go for businesses today, in states that protect against anti-gay discrimination.

Douthat disagrees, in another piece called “The Terms of Our Surrender.” He argues again that requiring the New Mexico photography studio to serve a gay couple over its religious objections means the end of “carving out protections for dissent.” It’s like saying “you bigots don’t get to negotiate anymore.” And Douthat can call on Andrew Sullivan here, who has said he doesn’t want photos or cakes or flowers from a business that doesn’t want him as a customer. “We’re not talking logic here,” Sullivan says. “We’re talking religious conviction. My view is that in a free and live-and-let-live society, we should give them space.”

The pragmatist in me feels the same way. As Sullivan and Douthat both argue, the gay-rights side is winning. What harm does it really do if a few businesses insist on holding out?

If this were just about private lawsuits, I could live with that. But it’s not. It’s also about the responsibility of a state to enforce its own anti-discrimination laws. Here’s what the Washington attorney general recently wrote to a florist that refused service to a gay couple: “It is my job to enforce the laws of the state of Washington. Under the Consumer Protection Act, it is unlawful to discriminate against customers based on sexual orientation.” Exactly. Remember, this kind of discrimination is still lawful in plenty of states. But where it is not, businesses shouldn’t be allowed to flout the rules of their own communities. That’s my problem with Douthat’s argument. It’s what the New Mexico case comes down to—and why the Supreme Court was right to stay out of it.