When I go back home to Ohio, the reason my marriage is legally recognized is Supreme Court Justice Anthony Kennedy.

On the Court, Kennedy was a champion for LGBTQ rights across the US. He was by no means perfect — but as the legal recognition of my marriage can attest to, he was a key figure. And his announcement today that he’s retiring could move a court that has been closely divided but largely favorable to LGBTQ rights to one that is outright hostile to equality once President Donald Trump and a Republican-dominated Senate, both opposed to LGBTQ rights, vote on Kennedy’s replacement. That could, in effect, start to undermine much of Kennedy’s LGBTQ legacy.

Kennedy was typically the swing vote or one of the swing votes in the Court’s 5-4 or 6-3 decisions for LGBTQ rights. Though he was in many ways a conservative justice (one nominated by President Ronald Reagan), he not only acted as a swing vote but wrote the majority opinions that established the laws of the land to protect LGBTQ people. Here are the big examples:

In 1996’s Romer v. Evans (decided 6-3), Kennedy invoked the 14th Amendment’s Equal Protection Clause to overturn a Colorado state constitutional amendment that prohibited local jurisdictions from protecting LGBT people from discrimination based on sexual orientation.

In 2003’s Lawrence v. Texas (6-3), Kennedy cited the due process clause’s guarantee of personal liberty to strike down Texas’s — and other states’ — ban on sex between gay men or lesbians.

In 2013’s United States v. Windsor (5-4), Kennedy ended the federal ban on marriage between same-sex partners.

In 2015’s Obergefell v. Hodges (5-4), Kennedy struck down state bans on the marriage of same-sex couples.

The Obergefell case, which originated in Ohio, in particular has had a huge effect on my life — giving my union legal recognition in the state where I lived for most my life. The decision is really worth reading in full. But if you want the gist of it, here’s the powerful conclusion:

No union is more profound than marriage, for it embodies the highest ideals of love, fidelity, devotion, sacrifice, and family. In forming a marital union, two people become something greater than once they were. As some of the petitioners in these cases demonstrate, marriage embodies a love that may endure even past death. It would misunderstand these men and women to say they disrespect the idea of marriage. Their plea is that they do respect it, respect it so deeply that they seek to find its fulfillment for themselves. Their hope is not to be condemned to live in loneliness, excluded from one of civilization’s oldest institutions. They ask for equal dignity in the eyes of the law. The Constitution grants them that right.

There have been some disappointing moments for LGBTQ advocates as well. This month Kennedy ruled in favor of Masterpiece Cakeshop in a case about whether the Colorado bakery could refuse service to a same-sex couple on religious grounds.

Still, the ruling was so narrow — focused on the process and the way that Masterpiece Cakeshop had been treated, and not so much on whether anti-LGBTQ discrimination is permitted under the law — that it isn’t likely to set a dangerous precedent for LGBTQ rights in the US. As Kennedy wrote in the majority opinion, “The outcome of cases like this in other circumstances must await further elaboration in the courts, all in the context of recognizing that these disputes must be resolved with tolerance, without undue disrespect to sincere religious beliefs, and without subjecting gay persons to indignities when they seek goods and services in an open market.”

In short, the ruling doesn’t create a license for anti-LGBTQ discrimination but instead asks the Colorado Civil Rights Commission to take care to respect people’s religious views in its decisions.

And by and large, this is just one of few dark spots in an otherwise exemplary record for LGBTQ equality.

Anti-LGBTQ advocates are already gearing up to weaken LGBTQ rights in the US now that Kennedy is leaving the Court. The anti-LGBTQ Alliance Defending Freedom quickly sent out a statement saying it “looks forward to the president’s nomination of a person to replace Justice Kennedy on the Supreme Court who will uphold the First Amendment and the original meaning of the Constitution.” And Republicans hostile to LGBTQ rights now have a chance to implement their anti-LGBTQ agenda.

The Kennedy exit is coming at a big moment for LGBTQ rights. There are some major issues that likely to arrive before the Court in the next few years: Do religious rights allow people to bypass laws that prohibit anti-LGBTQ discrimination in the workplace, housing, and public accommodations (such as hotels, bakeries, restaurants, and other venues that serve the public)? Does federal law already prohibit anti-LGBTQ discrimination in some settings? Can transgender people be forced to use a bathroom that doesn’t align with their gender identity?

Kennedy was a reliable swing vote toward LGBTQ rights. With his retirement, the future of that movement — at least when it comes to Supreme Court decisions — looks much less certain.

For more on what Kennedy’s departure from the Court means, read Dylan Matthews’s in-depth explainer.