The way Trump talks may convince some women he is not fit to act as commander in chief. Asma Hasan, a Republican woman living in Denver, Colorado, who is enthusiastic about voting for Clinton, finds it “hard to imagine voting for Trump when you consider the things he’s said about women, about minorities, even kicking a baby out of his rally.” It’s not just that his remarks are so often offensive, it’s that she believes Trump’s willingness to casually say shocking things could be dangerous. “If you’re going to be president, you need to think about what the perception will be of what you’re saying, how it will come across,” she said. “That’s especially important for a world leader.”

However Republican women vote in November, Trump could do lasting damage to the party as it attempts to win over women in future elections. “I think the party is losing an entire generation of voters, especially young women,” said Jennifer Pierotti Lim, the founder of an organization called Republican Women for Hillary. If Trump wins, even anti-Trump Republican women who want to remain in the party may find it hard to do so. “If Trump is elected and we become the party of Trump, and our leaders still aren’t able to push against him in terms of policy and rhetoric, then I can’t really see myself being part of a party like that,” Pierotti Lim said.

At least some Republican women already plan to leave the party. Over the course of the election, Suzanne Alvarez, a self-described lifelong Republican who lives in Spokane, Washington, decided she couldn’t support a party that won’t disavow Trump. “I was shocked when I saw the leadership in the party standing by him,” Alvarez said in an interview. “They helped create Donald Trump by not confronting the racism and paranoia that was infecting the party. By their silence, they validated Trump, and for me that is worth leaving the party over.” Alvarez plans to vote for Clinton, and she wants to become a Democrat as a result of Trump’s rise.

Not everyone wants to abandon the GOP, though. “I think the best outcome for the country and the Republican Party is for Trump to lose, and I want to be part of rebuilding the party along conservative principles when that happens,” said Kori Schake, an advisor to John McCain during the 2008 presidential campaign who plans to vote for Clinton. “I think it would take a lot for me to leave,” said Milloy, the chair of Republican Women for Hillary. “I believe we need voices of reason to help bring the party back to a point where it can be nationally successful.”

Anti-Trump Republican women may be called traitors and sellouts for now. But part of their motivation for speaking out may be a hope that they will be well-positioned to pick up the pieces of a broken party if Trump loses in November. If Republican leaders who endorsed Trump end up discredited, conservatives who forcefully opposed Trump from the beginning could prove influential in deciding how the party moves forward.