Washington (CNN) If Brett Kavanaugh is confirmed to the Supreme Court, the question that would hang over him, based on all he has been through and all he has said, is: Can he be fair?

Retired Justice John Paul Stevens, prominent law professors, attorneys and lawmakers have all raised that question as senators prepare to vote on his nomination Saturday.

Stevens told a Florida audience that there was "merit" to the criticism that Kavanaugh has "demonstrated a potential bias involving enough potential litigants before the court that he would not be able to perform his full responsibilities."

Kavanaugh, a 12-year veteran of an influential US appeals court in Washington, told the Senate Judiciary Committee last week that the sexual assault allegations of Christine Blasey Ford arose from "a calculated and orchestrated political hit, fueled with apparent pent-up anger about President Trump and the 2016 election ... revenge on behalf of the Clintons and millions of dollars in money from outside left-wing opposition groups."

The former aide to President George W. Bush who also served with independent counsel Ken Starr's investigation of President Bill Clinton, also declared, "As we all know, in the United States political system of the early 2000s, what goes around comes around."

Even from the start, President Donald Trump's nomination of Kavanaugh was incendiary. The atmosphere in the nation's capital and throughout the country is far more partisan than in earlier decades, and Kavanaugh would succeed centrist conservative Justice Anthony Kennedy, who held the key vote on the nine-member bench for abortion rights and gay marriage, to name just two of the many social policy issues Kennedy controlled.

Kavanaugh's partisan outbursts , suggesting he would not be a neutral arbiter, also may have reinforced public perceptions that the ideologically divided justices cast their votes based on political allegiances. Chief Justice John Roberts, also appointed by a Republican president, has previously expressed concerns that the public believes any nominee who emerges from a politically divisive confirmation process necessarily would be political.

On Thursday night, Kavanaugh published an essay in the Wall Street Journal acknowledging he may have gone too far.

"I was very emotional last Thursday, more so than I have ever been. I might have been too emotional at times. I know that my tone was sharp, and I said a few things I should not have said," he wrote.

Suspicions of impartiality are not new, but the nature of the accusations and Kavanaugh's angry, partisan response put this episode in a category of its own.

Other Supreme Court nominees -- in recent decades William Rehnquist and Clarence Thomas -- faced accusations, questions of veracity, and claims they would not be impartial jurists. Both were confirmed (Rehnquist as an associate justice in late 1971 and as chief justice in 1986) and Thomas (as an associate justice in 1991), but controversy over their nominations faded but never fully disappeared.

When do justices recuse themselves?

Questions of judicial fairness arise in two ways: broadly regarding a jurist's temperament, as is in the current dispute, and specifically on cases that come before the nine.

Regarding the latter, federal law says that any judge must disqualify himself or herself in any proceeding in which his or her "impartiality might reasonably be questioned." The Supreme Court is not covered by that mandate, as are lower court judges, but the individual justices have said they abide by it.

Rarely, however, do they recuse themselves from cases. They pull out, typically, if they have a clear financial interest, for example, with stock in a company that has been sued, or if they previously handled the case as a lawyer or lower court judge. It is exceedingly rare that a justice decides not to participate based on a substantive view on the issue.

New York University Law Professor Stephen Gillers, who specializes in legal ethics, said "the bar is very high" for motions that a jurist recuse himself or herself. He said case precedent requires specific evidence of bias for a specific situation.

Gillers said it was unlikely that Kavanaugh, if he were confirmed, would receive many motions from litigants for recusal. Yet, he said that would not be the end of the matter in the public eye.

"The fact that he's not going to be recused in politically sensitive cases doesn't mean that the losers in those cases will not believe that he was biased against them, Gillers said. "There's a lot of disconnect between the high bar of recusal and what the public believes" of a judge's prejudices and preconceptions.

More than 2,400 law professors from more than 190 schools delivered a letter on Thursday to the Senate questioning Kavanaugh's judicial temperament, based on his testimony in the second round of Senate hearings, when Kavanaugh defended himself against claims that he assaulted Ford when they were teenagers in Maryland. Even before that, some legal scholars asserted Kavanaugh would have to step aside in cases involving presidential power, based on Kavanaugh's prior statements in the area.

Stevens, nominated by Republican Gerald Ford and who served 1975-2010, drew a line between Thomas' statements and Kavanaugh's, concluding that, "there's nothing that Clarence did in the hearings that disqualified him from sitting in cases after he came on the court."

Thomas, who also served on the DC Circuit where Kavanaugh now sits, was accused of sexual harassment by Anita Hill, with whom he had worked at the Department of Education and Equal Employment Opportunity Commission. He had been nominated to succeed liberal legend Thurgood Marshall, the nation's first African-American justice.

"This is a circus," Thomas said at a 1991 hearing before the Senate Judiciary Committee. "It is a national disgrace. And from my standpoint as a black American ... it is a high-tech lynching for uppity blacks who in any way deign to think for themselves, to do for themselves, to have different ideas."

After Thomas was confirmed, lawyers did not regularly seek to remove from cases. He has been broadly regarded as a hard-right jurist who, even if bitter over the Hill claims, votes based on his ideology not on payback.

During Rehnquist's 1971 hearings to be an associate justice and then in 1986 to be chief justice, Rehnquist was accused of numerous incidents of racial bias, including of intimidating black voters at the polls in the 1960s when he was a lawyer in Phoenix. Rehnquist denied the allegations.

Among the witnesses who testified against Rehnquist in 1986 about alleged poll-watching activities in Arizona was James Brosnahan, a former assistant US attorney who had been investigating voter complaints.

Rehnquist was confirmed for elevation to chief, and afterward, he recused himself from Brosnahan's cases.