Despite this year’s Supreme Court’s ruling striking down bans on same-sex marriage, LGBT leaders are warning that a backlash against gay rights is coming.

The Court’s ruling, which guarantees gays and lesbians to right to wed, has been met with efforts to protest those protections. In Kentucky, for instance, a county clerk was jailed for refusing to issue marriage licenses to same-sex couples—and then celebrated as a spokeswoman for Christian conservatives. In Alabama, adoption rights for same-sex couples are at risk. And, in Houston, voters repealed a non-discrimination measure.

“Next week, we will celebrate seven months since we won the right to marry. Seven months. It’s like the blink of an eye,” National Center for Lesbian Rights Executive Director Kate Kendell said, warning that reactions to similar victories for civil rights came with violence. “We are in the middle of a full-throated backlash. … We are just seeing the beginning on the backlash. It will get worse before it gets better.”

Get our Politics Newsletter. The headlines out of Washington never seem to slow. Subscribe to The D.C. Brief to make sense of what matters most. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Indeed, a cautious tone is running through the annual Gay and Lesbian Victory Institute’s annual leadership summit, taking place on the Las Vegas Strip. Throughout panels and interviews, many of the advocates and elected officials expressed caution that the win on marriage was not the end of the fight for equality. They pointed to the Houston vote, which made the city the largest in the country where it is now legal to discriminate based on gender identity.

“Before the balloons even deflated, our opponents started rolling out really aggressive anti-LGBT policies,” said Danielle Moodie-Mills, the CEO of Politini Media.

Despite the historic victory on marriage, the summit was hardly jubilant. If anything, the participants were more alarmist than before.

“Now, with marriage equality, you can get married on Saturday and in the majority of the states on Monday be fired simply because of who you love,” said Rep. Mark Pocan, a Wisconsin Democrat. “We’ve had a big victory. … But we saw what just happened in Houston.”

Houston Mayor Annise Parker, who is a lesbian, acknowledged the setback in her city. “The average age of the voters in that race? 68 years old,” she said. “They defeated it by demonizing transgender women.”

In fact, the opponents of the non-discrimination measure made the vote about allowing men into women’s restrooms. “If you haven’t seen the ads, you should definitely see them. They will turn your stomach,” Kendell said. “That is so not a new tactic. Every gay man of a certain age who knows it was you who were pedophiles who were preying on children. That was only 20 years ago, 25 years ago.”

The Victory meeting, which promotes gay rights but is not explicitly partisan, urged leaders, donors and activists to keep involved. The organization now counts 450 openly gay elected officials in the country.

“You just can’t quit once you get a big victory,” said Gautam Raghavan, vice president at the Gill Foundation, which funds gay rights battles. “I worry that our straight allies are thinking our work is over. … I worry about funding. I worry that a lot of folks will start writing checks to something else. We can’t do that.”

Indeed, 33 states still allow discrimination against LGBT residents.

“Clearly, this year has been an incredible year for the LGBT movement,” said Kris Hayashi, the executive director of the Transgender Law Center. “The reality is the majority of the transgender community is struggling to survive on a daily basis. In every city, town, state, there are transgender people who are facing harassment, discrimination, violence and abuse.”

That threat sparked fear that these leaders channeled. “Cultural victories are often met with political backlash,” said Aisha Moodie-Mills, the President and CEO of Victory. “Make no mistake: Our opponents are gunning for us. … We have to fight back. We can’t just hope that the hatemongers are going to be drowned out by love.”

Read Next: Why the Next Gay Rights Push Will Be Different

Write to Philip Elliott at philip.elliott@time.com.