Since my ruling came down from the Supreme Court three years and two days ago in favor of same-sex marriage nationwide, I have officiated many weddings. In every ceremony, I have quoted from the last page of the opinion written by Justice Anthony Kennedy. I know thousands of other couples across the nation — gay, lesbian, straight — have used it in their ceremonies, too. People have it hanging on their walls. It begins by stating that “No union is more profound than marriage,” and it goes on to explain that people like me and my late husband John — who wanted my name to appear on his death certificate as his surviving spouse — were simply asking for “equal dignity” in the eyes of the law when we sought to have our union recognized by our government.

The opinion also says that the Constitution grants us that right. But now that Justice Kennedy has announced his retirement, it’s hard not to be afraid that it is going to be taken away.

I am not Justice Kennedy, and we can never truly understand another person. But every person thinks about the legacy that they leave behind for their family, their loved ones. Politicians think about the legacy they’re leaving behind for their constituents, the nation. And every judge thinks about the legal legacy they will leave behind. I struggle to understand how Justice Kennedy can look at our current environment and retire, knowing that his legacy of compassion and dignity, not to mention the civil rights of millions of people, are in serious risk.

Kennedy was the swing vote on not just my case but also decisions like Lawrence v. Texas and Windsor v. the United States, landmark rulings for LGBTQ Americans. To say I feel it in the pit of my stomach now that he is leaving the bench is an understatement. I feel despondent. I worry about the Trump Administration’s ability to nominate — and force through — another extremely conservative justice like Neil Gorsuch. And I’m not just worried about the LGBTQ community. I’m scared for women’s rights, for voting rights. I’m scared for every marginalized group in this nation who relies upon the Supreme Court to live up to those four words etched into the building’s western pediment: “Equal Justice Under Law.”

Opponents of marriage equality are going to waste no time doing everything they can to get a case into that building again to try to take that constitutional right away from us. I try to remind people that we might now have the right to marry, but in 31 states, a couple who exercises that constitutional right can be fired or can lose their home. And based on everything the Trump Administration has done, I believe the President is determined to take rights away, to take us back to the 1950s — if not further.

Not a day goes by that I’m not filled with dread by something that comes out of this Administration. It has expressed support for those who would use religion to refuse service to LGBTQ people, breathing life into “religious refusal” bills that are designed to elevate one person’s interpretation of their holy book above another’s in the public sphere. These efforts have nothing to do with law. They’re purely about telling us we don’t matter.

This scares me, because we’ve been down this road before. Religious beliefs were used to justify laws against interracial marriage. Religious beliefs were used to justify keeping the races separate. And that’s the antithesis of religious freedom, a constitutional right that the Supreme Court is in charge of interpreting for the rest of us.

Before his retirement announcement, as a member of the LGBTQ community, I saw Kennedy as something of a hero, someone who looked out for us, someone who used compassion in his interpretation of the Constitution. Three years ago, after he read the ruling in my case, I walked out of the Supreme Court and for the first time in my life, as an out gay man, I felt like an equal American. I felt like our nation was actually living up to its ideals. Many LGBTQ Americans have told me they felt the same way that day.

There are times I have to remind myself when I see it in print, when I hear it, that I’m that Obergefell. Because it still seems surreal. The thousands upon thousands of interactions I’ve had with people — on the street, in airplanes, everywhere I go — telling me stories, showing me photos, hugging me, crying. They explain how they finally feel safe as couples and as families. They tell me about how their concern about their children evaporated. They talk about how they’ve been together 40 years but feel so much more complete now that they have been married for three. Everything that ruling has meant to people across our nation, that’s what is at risk.

Every step forward in civil rights, there is always a backlash. That’s how it goes. But with Kennedy’s retirement, I am concerned that the backlash is going to turn into a complete ripping away of every bit of equality — every bit of progress — we’ve made over these decades. On Decision Day in my case, I also thought about my late husband John. I thought, You can finally be at peace. But at this point, I’m not so sure how much peace he can possibly be enjoying.

Sign up for Inside TIME. Be the first to see the new cover of TIME and get our most compelling stories delivered straight to your inbox. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Contact us at letters@time.com.