Win McNamee/Getty Images Brett Kavanaugh has a lifetime seat on the U.S. Court of Appeals for the D.C. Circuit.

If Brett Kavanaugh doesn’t reach the Supreme Court because of sexual assault allegations, he’ll still get to serve on what’s considered the second-most important court in the country for the rest of his life.

But if Democrats retake control of Congress, he may not get to return to that prestigious position so easily.

Christine Blasey Ford has accused Kavanaugh of sexually assaulting her at a house party when they were teenagers. The two of them are tentatively scheduled to testify before the Senate Judiciary Committee on Thursday.

On Sunday night, The New Yorker published a piece with allegations by a second woman, Deborah Ramirez, who said Kavanaugh exposed himself to her when they were students at Yale. The Supreme Court nominee has denied both claims.

“This alleged behavior would render Judge Kavanaugh unfit for the Supreme Court, or for any court,” said Emily Martin, vice president for education and workplace justice at the National Women’s Law Center.

Kavanaugh’s confirmation is increasingly in doubt, a sharp turn from what initially seemed like a smooth path to the Supreme Court. But even if Kavanaugh doesn’t get confirmed to the nation’s top court, he will remain a judge on the U.S. Court of Appeals for the D.C. Circuit, an incredibly powerful body that’s just a step below the Supreme Court.

If you stand with survivors, the path forward is clear: Brett Kavanaugh should be impeached ― not promoted. Heidi Hess, Credo Action

On Sunday, former New Jersey Gov. Chris Christie (R) lamented that Kavanaugh’s reputation will forever be tarnished, saying it will be very hard for him to return to the D.C. Circuit if he doesn’t get onto the Supreme Court.

“You are taking his career, and the idea that this guy could in fact go back to the court of appeals and continue to serve after this if he were not confirmed, I think is ― is really unlikely,” Christie said.

But if the allegations are serious enough that they tank Kavanaugh’s Supreme Court confirmation, there will be pressure to remove him from his current position as well.

“If you stand with survivors, the path forward is clear: Brett Kavanaugh should be impeached ― not promoted,” said Heidi Hess, co-director of the progressive group Credo Action.

Brian Fallon, executive director of the anti-Kavanaugh group Demand Justice, similarly said that Kavanaugh is not fit to hold a judgeship at any level.

“If and when they regain power, Democrats need to resume the full investigation into Kavanaugh’s lying and alleged sexual abuse that Republicans have so far refused to conduct,” he said.

Rep. Ted Lieu (D-Calif.) also said that if the allegations against Kavanaugh are true, the House Judiciary Committee “must open an investigation into these allegations to determine whether Kavanaugh should keep his current job or whether there are grounds for impeachment.”

The D.C. Circuit judge has also been a lecturer at Harvard Law School since 2005, and a group of students there are calling on the university to reconsider whether he should automatically return in January to teach a class.

Kavanaugh issued a statement on Monday denouncing the “smears” he has faced, calling them part of a “grotesque and obvious character assassination.”

“I will not be intimidated into withdrawing from this process,” he said. “The coordinated effort to destroy my good name will not drive me out. The vile threats of violence against my family will not drive me out. The last-minute character assassination will not succeed.”

The chief judge of the D.C. Circuit is Merrick Garland, the man whom President Barack Obama nominated to replace the late Supreme Court Justice Antonin Scalia and whom Republicans blocked from ever receiving a hearing or a vote.