A new worry in conservative circles has slowly gathered steam since the Supreme Court heard arguments in Obergefell v. Hodges, on whether the Constitution requires states to issue marriage licenses to and recognize the out-of-state marriages of same-sex couples. During oral arguments, Justice Samuel Alito asked Solicitor General Donald Verrilli whether the tax-exempt status of religious colleges and universities who opposed same-sex marriage would be in jeopardy should the Court hold for the plaintiffs. Verrilli hedged, saying he’d have to know more details to answer the question, but conceded that that it would “be an an issue.” That was enough for conservatives to start raising alarm bells about the Internal Revenue Service (which they already hate) yanking the tax exemption of religious organizations and churches who oppose same-sex marriage.

Travis Weber, director of the Family Research Council’s Center for Religious Liberty, told Fox News’ Shannon Bream that “he believed Verrilli’s logic would eventually extend to churches and just about ‘anyone holding a traditional view of marriage.'”

The American Family Association’s One News Now claimed that “Christian colleges and universities would be forced to decide between compromising their biblical beliefs and surviving financially,” and that churches and other institutions could be forced to shutter as well, because losing the tax exemption would spell financial ruin.

“Religious non-profits, Plan Now for Tax-Exemption Battles,” warned The Federalist.

This anxiety is not new. Last year, as the lower courts were striking down states’ bans on same-sex marriage (in the cases that ultimately went up to the Supreme Court), I wrote about a similar argument conservatives were making, raising fears that churches would lose their tax-exempt status if they refused to perform same-sex marriage ceremonies. As I noted then, the religious right was galvanized in the 1970s by the IRS’s revocation of the tax-exempt status of Bob Jones University over its policy banning interracial dating. That revocation–upheld by the Supreme Court–was a result of an IRS rule that permits the agency to deny or revoke a non-profit’s tax-exempt status if it violates a fundamental public policy (in that case, public policy against race discrimination).

The Bob Jones case is an essential part of the conservative movement’s institutional memory of what they consider to be government overreach, particularly by the IRS, and undue government interference with religious affairs and religious freedom. As historian Randall Balmer has documented, Bob Jones University’s loss of its tax-exemption mobilized the religious right more than abortion.

At last month’s arguments, Alito raised the Bob Jones case specifically:

JUSTICE ALITO: Well, in the Bob Jones case, the Court held that a college was not entitled to tax-exempt status if it opposed interracial marriage or interracial dating. So would the same apply to a university or a college if it opposed same-sex marriage? GENERAL VERRILLI: You know, I don’t think I can answer that question without knowing more specifics, but it’s certainly going to be an issue. I don’t deny that, Justice Alito. It is going to be an issue.

I reached out to two legal experts–one an expert on tax-exempt organizations, the other an expert on constitutional law–for insight into what kind of issue a ruling in the plaintiffs’ favor would really pose.

Richard Schmalbeck, a tax expert at Duke Law School, said the principle in Bob Jones has “not been applied broadly,” noting, for example, that it hasn’t been extended to deprive all-women’s colleges of their tax-exempt status. Caroline Mala Corbin, a professor at the University of Miami Law School, reiterated what she told me last year: that if the IRS and the courts had not applied the Bob Jones principle to revoke the tax exempt status of organizations that discriminated based on gender, it seemed unlikely that they would extend this principle to sexual orientation discrimination. What’s more, she said, federal law bans race discrimination, demonstrating a broad consensus to protect as fundamental public policy. Given that discrimination based on sexual orientation has not been banned by any federal law, we are far from such a consensus on sexual orientation discrimination. And even though federal law bans discrimination based on gender, the Bob Jones principle has never been extended to tax-exempt organizations that discriminate based on gender.

Think about it this way: there are religious denominations that ban women’s ordination, and still have tax-exempt status.

Schmalbeck pointed out that how the Supreme Court’s ruling in the marriage cases would impact the tax-exempt status of organizations that oppose same-sex marriage would depend on the specifics and the breadth of the holding. If, for example, the Court ruled that “states cannot withhold the benefits of marriage on grounds of sexual orientation,” that would “not necessarily reach religious institutions exercising their rights to worship . . . and believe in the things that their doctrines hold. The Court could even drop a footnote saying that they weren’t deciding that case, and that no such inference should be drawn,” he said. But if the Court were to adopt “a more aggressive version of the idea that discrimination on the basis of sexual orientation is fundamentally abhorrent,” that could possibly “support a subsequent finding that a church that refuses the sacrament of marriage to a same-sex couple is violating public policy.” However, he said he thought such an outcome unlikely.

It’s worth thinking about this in the context of the way the IRS’s interaction with tax-exempt organizations, particularly religious ones, has functioned in practice. There hasn’t been another Bob Jones–not because religious organizations don’t refuse to serve or recognize religious beliefs inconsistent with their own, but because the IRS has given them wide latitude to do so. Every day religious organizations likely refuse to serve members of the public who do not adhere to their religious beliefs. As Justice Elena Kagan noted in the oral arguments, rabbis are free to refuse to marry couples if one member of the couple is not Jewish. That refusal is protected by the First Amendment, even though the couple, too, has religious freedom rights.

What did Verrilli mean when he said, “it is going to be an issue”? Perhaps he meant opponents of same-sex marriage would make it an issue. Not that the actual loss of tax-exempt status would be an issue.