With Justice Anthony Kennedy retiring, the man President Trump once called an "absolute disaster" potentially is poised to inherit the mantle of "swing" justice – and solidify his power on the Supreme Court.

Whether Chief Justice John Roberts can, or wants to, fill that vacuum is the big question in judicial circles right now.

"If the president, as everyone expects, nominates a conservative to fill Justice Kennedy's seat, the spotlight will shift to the chief justice, who will be viewed, rightly or wrongly, as the swing justice," said Thomas Dupree, a former top Bush Justice Department official. "In many ways, I think this would be a role the chief justice would welcome. He's played this role before.”

While Roberts is chief justice, Kennedy has long held the power. He holds the reputation, deservedly or not, of the man in the middle, whose votes on a range of hot-button issues proved decisive over a 30-year high court career.

But the President George W. Bush-appointed Roberts has flashed his own independent streak over the years, including on ObamaCare, and could be the closest to a wild card of those left on the bench. Kennedy hated the title of "swing" justice, and Roberts too may come to rue the perception that his future votes on the court are up for grabs. Both men are conservatives, nominated by Republican presidents. Both see themselves as institutional stewards of the court, particularly as an insular body free of partisan politics.

DEMS CALL TO DELAY KENNEDY SUCCESSOR VOTE -- WILL IT BACKFIRE?

But for all the Democratic warnings that Kennedy’s retirement ensures an unflinching conservative majority, conservatives have their own concerns. The president will have control over the person to fill Kennedy's seat, but not the mind of the chief justice.

Some right-leaning legal activists have long believed Roberts is not someone whose vote they can take for granted, despite the Republican-appointed 63-year-old chief justice having a mostly conservative record for 13 years.

"It's the Roberts court now," said Michael Dorf, a Cornell law school professor who also runs the dorfonlaw.org legal blog. "The new justice will certainly be at least as conservative as CJ Roberts, so regardless of how this goes, the chief justice is now the center of the court."

Those close to Roberts told Fox News he has been especially sensitive to political attacks on the court as an institution and its role as a fair arbiter of the rule of law.

The "chief," as he is informally known around the court, has publicly stated his long-term efforts to forge consensus with his colleagues whenever possible, show respect for precedent and preserve the court's reputation.

In a 2014 speech, he said the "partisan rancor" among members of Congress "impedes their ability to carry out their functions. I don't want this to spill over and affect us.”

'It's the Roberts court now.' — Michael Dorf, Cornell law school professor

This was after Roberts' dramatic 2012 deciding vote upholding the key funding provision of the Affordable Care Act -- saving then-President Barack Obama's signature domestic achievement.

Conservatives called it a calculated act of betrayal, a narrative Trump echoed on the campaign trail.

"Justice Roberts really let us down," candidate Trump said at a December 2015 campaign rally. "What he did with ObamaCare was disgraceful, and I think he did that because he wanted to be popular inside the Beltway."

In other speeches, Trump has used words like "disappointment" and "nightmare." He had cited the late Justice Antonin Scalia and Justice Clarence Thomas as his favorites.

Yet as president, Trump heaped praise on moderate-conservative Kennedy following his retirement notice, meeting privately with the 81-year-old justice at the White House Wednesday just after the news was revealed.

For Trump, how the new court rules is nothing less than a matter of legacy.

Since the start of his term, the White House has been aggressively filling dozens of lower-court vacancies with conservatives in the Trump mold, knowing this would ostensibly have the greatest influence on the broader rule of law. Since the high court gets the final word on legal issues big and small, the president can only hope his new nominee – along with Roberts – will stick with fellow conservatives.

Elizabeth Wydra, president of the progressive Constitutional Accountability Center, views those like Roberts as a critical check on Trump’s potential excesses.

"The judicial branch will be an important check in the next few years,” she said. “And they can serve as a check, especially if there are threats to constitutional structure."

Some liberal leaders hold out tenuous hope Roberts takes a more active role as that "swing" vote – but there is conflicting evidence he would drift left with abandon.

Roberts wrote the 2015 dissent when the court struck down state laws banning same-sex marriage. He said the 5-4 majority's conclusion -- authored by Kennedy -- of a nationwide right guaranteed by equal protection and due process was misguided.

In language many social conservatives applauded, Roberts said the opinion was about political will, not legal judgment.

"The court invalidates the marriage laws of more than half the states and orders the transformation of a social institution that has formed the basis of human society for millennia," he said. "Just who do we think we are?"

And in recent weeks, Roberts was firmly in the conservative camp in two high-profile cases: siding with Trump on the so-called "travel ban” and in favor of a Colorado baker who cited his Christian views when refusing to make a wedding cake for a same-sex couple.

But scorn poured in from the right when Roberts last week sided with the four more liberal justices to rule against the administration on whether police need a warrant to obtain cell phone records. He cited the privacy rights of Americans over law enforcement.

Ten days before Scalia's 2016 death, Roberts was unusually candid making the case that courts should have a limited role, resisting the temptation to be the final arbiter of issues great and small. And he offered a prescient warning of sorts on the effects of a partisan fight over filling future Supreme Court vacancies.

"We don't work as Democrats or Republicans, and I think it's a very unfortunate perception that the public might get from the confirmation process," he said. "When you have a sharply political divisive hearing process, it increases the danger that whoever comes out of it will be viewed in those terms."

Then Roberts indirectly addressed suggestions then roiling in the presidential primaries that he and the court were engaging in "judicial activism” -- the ultimate dis.

"If we uphold a particular political decision, that remains the decision of the political branches, and the fact that it may lead to criticism of us is often a mistake," he said. "We do have to be above or apart from the criticism because we, of course, make unpopular decisions -- very unpopular decisions."