Until recently, almost all Americans were for religious liberty. Then LGBTQ rights and the contraception mandate came along. This week, the Supreme Court heard oral arguments over the question of whether a small cake-making business can refuse service to a gay couple — a clash both left and right have elevated into a key moment in the culture wars.

Many on the left now regard “religious liberty” as a coded term for discrimination and exclusion. Many on the right feel they are being persecuted for their unpopular beliefs.

The current unpleasantness is unnecessary. The tradition of religious liberty has the resources to handle this. Religious liberty has never meant a right to hurt people, but it has stood for a tolerance of the unfamiliar that opens a path to a reasonable settlement.

With only a slight course correction — aided, one hopes, by a Supreme Court that aims to dampen culture clashes, not exacerbate them — religious freedom can be everyone’s friend again.

Religious liberty is in many ways the primitive core of liberalism. It was the original basis for protecting a private sphere that government could not invade. “The one only narrow way which leads to Heaven is not better known to the Magistrate than to private Persons,” John Locke wrote in 1689, “and therefore I cannot safely take him for my Guide, who may probably be as ignorant of the way as myself, and who certainly is less concerned for my Salvation than I myself am.”

Since the colonial era, American law often gave religion special treatment when religious beliefs clashed with secular law. Quakers and Mennonites were excused from the draft even in those early years. Clergy could not — and still cannot — be forced to reveal in court what they learned in confession.

Trials used to be held on Saturdays, the Jewish Sabbath; Jews were not compelled to appear or testify. During Prohibition, Catholics were allowed to use sacramental wine. (The protection of religion was inconsistent, to be sure: The practices of slaves and Native Americans got no respect, and the federal government crushed Mormon polygamy.) None of this was thought to be required by the Constitution, but religious scruples were respected and often accommodated.

The right to a religious accommodation is not an absolute one

Often such accommodation doesn’t hurt anyone. Many laws, such as military conscription, taxes, environmental regulations, and drug laws will accomplish their ends even if there is some deviation from the norm they set forth, so long as that deviation does not become too great. Ideally, the law would accommodate all deep and valuable commitments to the extent that it can — and some scholars propose broadening exemptions to include secular convictions, too. “Deep commitments” is not, however, an administrable legal category. Religion is a workable proxy.

The right to religious exemption is, however, a peculiar kind of right. It is sometimes suggested that it’s in the nature of rights that they always override the ordinary weighing of costs against benefits. But a right to free exercise, if it includes any right to accommodation, is only a right to a certain kind of weighing: Religion is treated as a good that should be allowed to be pursued unless the marginal cost is too high. The right is a right to have that marginal cost considered in individual cases — a favor that is not done for nonreligious objectors to particular laws.

This balancing has traditionally been done by legislatures, not courts. The notion of a judicially enforceable right to free exercise was rejected by the Supreme Court in Reynolds v. United States (1878), in which a Mormon was prosecuted for polygamy. Reynolds argued that he had a right to his multiple marriages if they were entered into out of religious duty. The Court declared that this would make every person who invoked religious reasons a “law unto himself.” Reynolds’s proposed rule would protect human sacrifice, it noted.

The Court made judicial balancing of religious obligations and societal obligations a right, for the first time, in Sherbert v. Verner (1963), which involved a modest and harmless claim: A Seventh-Day Adventist, fired for refusing work on Saturdays, sought unemployment benefits. The state claimed that the religious objection was not a sufficient cause for refusing Saturday work. The Court disagreed. Its decision, by ultraliberal Justice William Brennan, held that exemptions were constitutionally required unless the state could show a compelling justification for denying them.

Justice Scalia’s game-changing 1990 opinion

That was a significant change in doctrine, but it hardly opened a Pandora’s box of religious exemption claims. Indeed, in the decades when the Sherbert rule was the law, 1963 to 1990, religious claimants won fewer than half of the exemptions they sought. In Employment Division v. Smith (1990), however, the Court abandoned that rule, in another case involving a seemingly innocuous, marginal issue: Native Americans wanted to participate in traditional peyote rituals.

The Court said they could not do so, because the laws banning peyote use applied equally to all citizens and did not single out any sect for discrimination. That decision was written by Justice Antonin Scalia, who didn’t mind special treatment of religion but worried about the discretion that Sherbert’s balancing test gave to the courts. Congress was nearly unanimous in reversing that result by statute: The Religious Freedom Restoration Act (RFRA) passed unanimously in the House and drew only three opposing votes in the Senate.

In 1997, the Supreme Court held that Congress had exceeded its constitutional powers by making the RFRA applicable to the states. (It remains valid as applied to federal law.) Scalia didn’t like balancing as a matter of constitutional law, but he and the other justices didn’t question Congress’s power to limit its own statutes. Congress couldn’t impose religious exemptions on state laws, but it could do so for federal laws.

Many state legislatures responded by passing their own state-level RFRAs. At first, these were not especially controversial either — on either side of the aisle. Barack Obama, as a state senator, voted for one of the earliest ones in Illinois in 1998. There are now 21 state RFRAs. Most of these came soon after the Court restricted the federal law, but interest in them waned. Only three were enacted between 2003 and 2013.

Even post-RFRA, courts have not been overzealous in accommodating religion. Religious claimants invoking the federal law still usually lose today — more than 70 percent of the time, according to one study.

Conservatives have pushed too far on two issues: contraception and same-sex marriage

So what happened to take us from the long period of consensus to today’s culture-war posturing? A couple of high-profile disputes have given religious freedom a bad name.

Before Obamacare, insurance often excluded coverage of medical needs specific to women, making women bear higher health care costs than men — as much as $1 billion a year more in the aggregate. The Affordable Care Act fixed that, including a requirement that employee and student health insurance plans cover the costs of most forms of contraception.

Certain churches and other nonprofit religious groups objected to facilitating what they regard as immoral conduct. The Obama administration devised accommodations for bona fide religious organizations that objected, but refused accommodations to for-profit businesses whose owners religiously objected to some or all of the mandated contraception coverage. Those businesses employed enormous numbers of women who did not share the employers’ religious beliefs.

The most prominent, the Hobby Lobby chain, has more than 13,000 full-time employees. The Supreme Court gave the store its exemption, but not the relief it was asking for, which was to absolutely cut off its employees from coverage for the expensive forms of contraception that the employer found objectionable.

In Burwell v. Hobby Lobby Stores Inc., the Supreme Court — applying the Religious Freedom Restoration Act — held that the Obama administration could extend to for-profit companies the religious nonprofit accommodation, in which insurers separately provided the coverage with their own funds. But note that a critical premise of the majority opinion was that all female employees and dependents of the company would in the end receive the mandated contraception coverage. “[T]he effect of the HHS-created accommodation on the women employed by Hobby Lobby … would be precisely zero,” the majority wrote.

The Trump accommodation has no such proviso. The administration shows the world how much it values religious liberty by knowingly bringing about additional unintended pregnancies, low-birth-weight babies, and abortions.

This is a dramatic departure from traditional understandings of religious freedom, which was always based on the premise that the exercise of one person’s religion should not hurt anyone else. In Thomas Jefferson’s classic formulation: “It does me no injury for my neighbour to say there are twenty gods, or no god. It neither picks my pocket nor breaks my leg.” But being forced to pay for contraceptives that the law says must be covered by insurance is exactly like having one’s pocket picked. And involuntary pregnancy is worse than a broken leg.

Beginning about 10 years ago, a few cases arose in which businesses refused to facilitate same-sex marriages by providing flowers or cakes, and were sued for discrimination.

The second wave of state-level “religious freedom” laws was less benign than the first

Recall that the movement for state RFRAs lost steam by 2003. In 2014, a new wave of such laws was introduced — and this time, conservatives clearly intended them to cover such discrimination (though, because they enacted a balancing test, it was far from clear that religious claimants would win). One of the earliest of these new-wave religious freedom laws, in Kansas, was essentially a blanket license to discriminate against same-sex couples. It was widely condemned, and the state Senate declined to take it up. The same month, Arizona’s legislature passed a state RFRA, but it too came across as a license to discriminate, and the governor vetoed it.

This struggle continues, most prominently in the Masterpiece Cakeshop case now being argued in the Supreme Court, where a religious baker is trying to shoehorn his claims into a free speech argument. The free speech argument won’t fly — it’s impossible to distinguish artistic cake making from ordinary food preparation in a workable way — but there is a discernible religious freedom claim.

Discrimination is a harder case than contraception. The law allows refusals of service when these are idiosyncratic, such as when a merchant won’t serve anyone who isn’t wearing a plaid shirt. (After all, others will provide the needed service.) The law intervenes when discrimination is pervasive. Would accommodating the occasional baker or florist unleash a flood of claims and defeat anti-discrimination protection? That would certainly have happened if there had been religious exemptions from the Civil Rights Act of 1964. No one knows whether that would happen with anti-discrimination protection for gay people.

Both issues — a desire not to pay for contraception and the “right” to refuse service to LGBTQ people — involve a very small number of claimants. But the religious right, deeply attached to a narrative of persecution, has trumpeted both, making this tiny subset of religious claims the only ones that many Americans hear about.

The problem in both the contraception and discrimination cases is not that there isn’t a plausible religious liberty claim; there is. The problem is that in these cases, the claim should be defeated by the rights of other people. That is hardly true of all religious liberty claims.

Muslims are the object of pervasive, ignorant hatred, made worse by Trump’s despicable propaganda. Absent religious liberty protections, many communities would have a free hand to manipulate zoning laws to bar the construction of mosques, and Muslim public employees could be forbidden to wear headscarves.

Or consider prison cases, which generally involve grooming and clothing, diet, group worship, and access to literature. The Supreme Court recently rejected a preposterous claim that Muslim prisoners could not be permitted to grow half-inch beards because they could hide weapons in them. Jewish prisoners are entitled to kosher food. Normally, prisons can invoke security to justify doing almost anything they want, however arbitrary; religious liberty claims help establish that prisoners remain human beings, with rights.

Compromises are possible. Obama was developing one for the contraception mandate, unsurprisingly more sober and balanced than Trump’s. Even with anti-LGBTQ discrimination, legislative deals might be possible: Perhaps businesses of five of fewer employees — genuine family businesses — could be freed to omit contraceptive coverage. Perhaps religious colleges could be allowed to exclude same-sex couples from married student housing. Utah managed to craft a deal that prohibited discrimination while protecting the autonomy of religious organizations.

The American political left has centrally been concerned, since the 1960s, with the right to be weird. It promised to free us from the traditional white male heterosexual template. In a way, the old idea of religious liberty was radicalized: Each person should be free to find his own way to salvation.

The idea that “religious liberty” is a mere cover for bigotry is misguided. The existing law of accommodation already contains within itself the resources to reject the claims that most trouble the left. The broader attack on religious liberty is inconsistent with the aspiration to have a society that is broadly hospitable to diversity.

Deep disagreement about moral fundamentals is nothing new; it is what religious diversity consists of. That ought to include disagreement about such fraught matters as sexuality. Moral disagreement about things that matter a lot is an inevitable consequence of a free society. The best we can hope for is to live peacefully together in mutual contempt.

Return the law of religious liberty to the reasonable equilibrium it had reached only a few decades ago, and we have a shot at achieving that goal.

Andrew Koppelman is the John Paul Stevens professor of law at Northwestern University, where he is also a professor (by courtesy) of political science. He is the author of Defending American Religious Neutrality.

The Big Idea is Vox’s home for smart discussion of the most important issues and ideas in politics, science, and culture — typically by outside contributors. If you have an idea for a piece, pitch us at thebigidea@vox.com.