Ever since election night 2016, right-of-center Americans who did not vote for Donald Trump or who did so grudgingly have done battle with our partisan identities. Along the way, we’ve lost a few associates, but also been joined by fresh faces.

The latest addition to this roster is James Comey, who recently said in an ABC News podcast that the GOP “has left [him]” and that he “can’t be associated with it.” He also said that “[Republicans] don’t represent anything I believe in... I see the Republican Party, as near as I can tell, reflects now entirely Donald Trump’s values. It doesn’t reflect values at all.”

Comey has joined the ranks of those who publicly destroyed their GOP registration cards in the course of 2016, and figures like The Washington Post’s Jennifer Rubin, whom Politico has described as “a leading voice for a group of conservative intellectuals who don’t fit comfortably in either political party.”

Many of these folks are friends and I sympathize with them. But I also deeply, deeply disagree with them.

If you ever watched The West Wing, you may remember Matt Skinner. He’s the gay Republican congressman who makes two pointed comments that should be at the crux of every Trump-opposing or Trump-skeptical right-of-center individual’s thinking about party affiliation. The first is, “I agree with 95% of the Republican platform… My life doesn’t have to be about being a homosexual. It doesn’t have to be entirely about that.” The second is, “I never understood why you gun control people don’t all join the NRA. They’ve got two million members, you bring three million to the next meeting, call a vote. ‘All those in favor of tossing guns?’ Bam. Move on.”

The point is this: A party is what you make it.

Trump has made the GOP his by slapping an “R” on his chest and attracting like-minded people to it. And people who were Republicans long before he was have gone along for the ride. While under 40 percent of Americans approve of Trump, something like 80 percent of Republicans approve of him, according to Gallup. His opposition to free trade has succeeded in flipping Republican voters’ views of trade on their historical head. The congressional GOP, which many Trump-dissenting (former) Republicans hoped would act as a counterbalance to him, often instead acts as a pushover, doing his bidding.

“ Why should James Comey or Jennifer Rubin ditch a party they’ve belonged to for longer than he has? ”

And yet the wrong lessons are being taken away here. Rather than seeing permanence in the GOP’s Trumpite shift, people should see the reality that political parties change—a lot. Just as the Democratic Party isn’t the party of Bill Clinton, the GOP is not the party of Reagan, or Eisenhower, or Goldwater, or Rockefeller, or Compassionate Conservative Bush 43—and those Republican Parties often differed a great deal from the others, apart from on maybe two or three core items.

Parties change. What the GOP is today under Trump is not likely to be what it is in even ten years. But that’s only the reality if people who want it to be something different in future stay in it, and work to change it from the inside. Instead, a whole bunch of them are taking a walk, goaded into doing so by people who rabidly disagree with them and know they’ll be in greater control, and more dominant, with the dissenters out in the wilderness, doing none of the things that actually get you power and influence in a political party.

And let’s be clear: These people aren’t likely to find an actual home in the Democratic Party—which is very unlikely to embrace core right-of-center principles anytime soon, though of course these folks could engage with them and try, just as they could with the GOP. They’re also not likely to stand up a third party that can prove electorally competitive or functional anytime soon. If we’re doing a simple mathematical equation assessing where one can most efficiently effect change, the answer still remains the GOP—even if the odds look really slim.

But there are ample precedents for engagement in the GOP yielding results critics. The most obvious one is the fight over LBGT rights.

Back in 2006, nary a single elected Republican expressed support for ditching Don’t Ask Don’t Tell or recognizing same-sex marriage. But a big part of why Don’t Ask Don’t Tell died a death and freedom to marry became the law of the land was because Republicans moved on the issue—because grassroots pro-LGBT Republican activists like me didn’t walk away, we doubled down.

Just before the Supreme Court Obergefell ruling on same-sex marriage, a Reuters/Ipsos poll found that 56 percent of Republicans would attend a same-sex wedding of a loved one, and about 40 percent of Republicans, including several prominent members of Congress, explicitly supported it. Before Don’t Ask Don’t Tell was overturned, you could find lots of Republicans, elected and not, who thought the policy was stupid. Today, we’re debating transgender troops and cake-baking. That is nothing short of a sea change, and proves that engagement works, whereas walking away is dicier.

It also feels silly, when you consider how long many of those ditching the GOP have been a part of it, relative to newcomers like Trump. Why should Comey or Rubin ditch a party they’ve belonged to for longer than he has? He’s the newcomer. But at the end of the day, if enough people “drop out,” it really will become Trump’s party, and not just for the duration of his presidency.

MSNBC host Chris Matthews is fond of saying that Democrats fall in love, Republicans fall in line. That’s what happened with Trump. Right up until when he locked down the nomination in the context of a quasi floor-fight at the 2016 convention, there was significant opposition to him. After that, Republicans mostly got on board, because it’s what we do.

If #nevertrumpers, or Trump skeptics, want to change the GOP, we need to understand what Matthews already does, and engage, not walk away—even if doing so is painful and frustrating and maddening.