It’s no secret that Justice Anthony Kennedy, who announced his imminent retirement on Wednesday, always had an eye toward his legacy. During this 30-year tenure on the Supreme Court, Kennedy often wrote grandiose opinions that seemed designed to be quoted and admired by future generations. That line of thinking was particularly evident in a series of decisions protecting gay Americans’ right to equal dignity. Yet by choosing to retire under President Donald Trump, Kennedy imperils that legacy, throwing his most celebrated and far-reaching decisions into serious jeopardy.

Kennedy’s gay rights opinions place three fundamental limits on the government’s ability to discriminate against gays. First, the government may not act out of mere “animus” toward sexual minorities. Second, the government may not outlaw sexual intimacy between individuals of the same gender. Third, the government may not refuse to license same-sex marriages or treat those unions differently from heterosexual marriages. Taken together, these rulings led gay Americans out of second-class citizenship, affirming their constitutional right to equal treatment by the state and federal government.

The cornerstone of Kennedy’s gay rights jurisprudence—Obergefell v. Hodges, the marriage case—was a 5–4 decision. Three dissenters—Chief Justice John Roberts, as well as Justices Clarence Thomas and Samuel Alito—are still on the court. One, Justice Antonin Scalia, died, but Trump replaced him with Justice Neil Gorsuch, who promptly expressed fierce hostility to gay equality. So there are already four justices who vigorously reject the constitutional right to same-sex marriage.

Trump is all but assured to replace Kennedy with a judge who shares the conservatives’ opposition to gay rights. (Brett Kavanaugh of the U.S. Court of Appeals for the District of Columbia Circuit is a top contender.) This justice can provide a fifth vote to begin cutting back Obergefell. State courts have already had a go at this. The Texas Supreme Court, for instance, refused to extend spousal benefits to married same-sex couples, while the Arkansas Supreme Court allowed the state to discriminate against same-sex parents. Ultimately, the U.S. Supreme Court reversed the Arkansas decision—but Gorsuch dissented, along with Thomas and Alito. Once Kennedy is replaced, a new conservative majority could chip away at marriage equality by embracing these sidelong attacks, then overrule it outright once they’ve unsettled its precedential basis.

If there is any hope of Obergefell surviving the Trump era, it lies with Roberts, who occasionally places institutional concerns over political and ideological preferences, and might be persuaded to affirm Obergefell to prevent chaos for same-sex couples across the country. Yet toward the start of his acrid Obergefell dissent, Roberts declared: “The majority’s decision is an act of will, not legal judgment. The right it announces has no basis in the Constitution or this Court’s precedent.” And at the end, he advised same-sex couples that they should not pretend like they deserved their victory. “Do not celebrate the Constitution,” he wrote. “It had nothing to do with it.” The man who wrote these words is now gay Americans’ last, best hope. His vote may be the only thing that stands in the way of states that wish to nullify all same-sex marriages.

Court-watchers have long wondered which part of Kennedy’s legacy the justice himself was most interested in preserving. Would he wish to save Citizens United, allowing unlimited corporate electioneering, and other landmark conservative rulings favoring federalism, gun rights, and Republicans? Or would he rather safeguard Obergefell and its predecessors by retiring under a Democratic president guaranteed to replace him with a gay-friendly justice?

On Wednesday, we learned the answer, and it’s a grim one for LGBTQ Americans. Their marriage rights were secure so long as Kennedy sat on the court with four liberals. In his absence, there is a very real chance that nationwide marriage equality will become a thing of the past.