Two weeks ago, with a decision in Obergefell v. Hodges on the way, Sen. Mike Lee of Utah introduced the First Amendment Defense Act, which ensures that religious institutions won’t lose their tax exemptions if they don’t support same-sex marriage. Liberals tend to think Sen. Lee’s fears are unwarranted, and they can even point to Justice Anthony Kennedy’s opinion in Friday’s case, which promises “that religious organizations and persons [will be] given proper protection.”

But I don’t think Sen. Lee is crazy. In the 1983 Bob Jones University case, the court ruled that a school could lose tax-exempt status if its policies violated “fundamental national public policy.” So far, the Bob Jones reasoning hasn’t been extended to other kinds of discrimination, but someday it could be. I’m a gay-rights supporter who was elated by Friday’s Supreme Court decision — but I honor Sen. Lee’s fears.

I don’t, however, like his solution. And he’s not going to like mine. Rather than try to rescue tax-exempt status for organizations that dissent from settled public policy on matters of race or sexuality, we need to take a more radical step. It’s time to abolish, or greatly diminish, their tax-exempt statuses.

Get The Brief. Sign up to receive the top stories you need to know right now. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

The federal revenue acts of 1909, 1913, and 1917 exempted nonprofits from the corporate excise and income taxes at the same time that they allowed people to deduct charitable contributions from their incomes. In other words, they gave tax-free status to the income of, and to the income donated to, nonprofits. Since then, state and local laws nearly everywhere have exempted nonprofits from all, or most, property tax and state income tax. This system of tax exemptions and deductions took shape partly during World War I, when it was feared that the new income tax, with top rates as high as 77%, might choke off charitable giving. But whatever its intentions, today it’s a mess, for several reasons.

First, the religious exemption has forced the IRS to decide what’s a religion, and thus has entangled church and state in the worst way. Since the world’s great religion scholars can’t agree on what a religion is, it’s absurd to ask a bunch of accountants, no matter how well-meaning. You can read part of the IRS’s guidelines for what’s a bona fide religion here; suffice it to say that it has an easier time saying what’s not a religion. The site gives the example of the rejection of an application from an “outgrowth of a supper club … whose primary activities were holding meetings before supper, sponsoring the supper club, and publishing a newsletter” but which professed a religious doctrine of “ethical egoism.”

On the other hand, the IRS famously caved and awarded the Church of Scientology tax-exempt status. Never mind that the Scientology is secretive, or that it charges for its courses; or that its leader, David Miscavige, lives like a pasha. Indeed, many clergy have mid-six-figure salaries — many university presidents, seven-figure salaries — and the IRS doesn’t trouble their tax-exempt status. And many churches and synagogues sit on exceedingly valuable tracts of land (walk up and down Fifth Avenue to see what I mean). The property taxes they aren’t paying have to be drawn from business owners and private citizens — in a real sense, you and I are subsidizing Mormon temples, Muslims mosques, Methodist churches.

We’re also subsidizing wealthy organizations sitting in the middle of poor towns. Yale University has an endowment of about $25 billion, yet it pays very little to the city of New Haven, which I (as a resident) can assure you needs the money. At the prep school I attended (current endowment: $175 million), faculty houses, owned by the school, were tax-exempt, on the theory that teachers sometimes had students over for dinner, where they talked about history or literature or swim practice.

Meanwhile, although nonprofits can’t endorse political candidates, they can be quite partisan and still thrive on the public dole, in the form of tax exemptions and deductions. Conservatives are footing the bill for taxes that Planned Parenthood, a nonprofit, doesn’t pay — while liberals are making up revenue lost from the National Rifle Association. I could go on. In short, the exemption-and-deduction regime has grown into a pointless, incoherent agglomeration of nonsensical loopholes, which can allow rich organizations to horde plentiful assets in the midst of poverty.

Defenders of tax exemptions and deductions argue that if we got rid of them charitable giving would drop. It surely would, although how much, we can’t say. But of course government revenue would go up, and that money could be used to, say, house the homeless and feed the hungry. We’d have fewer church soup kitchens — but countries that truly care about poverty don’t rely on churches to run soup kitchens.

Exemption advocates also point out that churches would be squeezed out of high-property-value areas. But if it’s important to the people of Fifth Avenue to have a synagogue like Emanu-El or an Episcopal church like St. Thomas in their midst, they should pay full freight for it. They can afford to, more than millions of poorer New Yorkers whose tax bills the synagogue and church exemptions are currently inflating.

So yes, the logic of gay-marriage rights could lead to a reexamination of conservative churches’ tax exemptions (although, as long as the IRS is afraid of challenging Scientology’s exemption, everyone else is probably safe). But when that day comes, it will be long overdue. I can see keeping some exemptions; hospitals, in particular, are an indispensable, and noncontroversial, public good. And localities could always carve out sensible property-tax exceptions for nonprofits their communities need. But it’s time for most nonprofits, like those of us who faithfully cut checks to them, to pay their fair share.

Read next: Episcopalians Vote to Allow Gay Marriage in Churches

Listen to the most important stories of the day

Contact us at letters@time.com.