Virginia Gov. Ralph Northam was clinging to his job Tuesday amid indications that he might not be forced out if he refuses to quit amid the firestorm over racist yearbook photos from more than three decades ago.

Northam, a Democrat, has emerged from staff and Cabinet meetings in recent days without addressing the calls for his resignation from both sides of the aisle. If he doesn't walk away, it would take a bipartisan effort to remove him.

Republican State House Speaker Kirk Cox said he has no plans to try to force Northam out. Cox called for Northam to resign, saying he no longer has the authority to lead the state, but he said Northam might not meet the constitutional provisions for impeachment, including misbehavior while in office or mental or physical incapacitation.

Want news from USA TODAY on WhatsApp?Click this link on your mobile device to get started

"Nothing that has happened so far is grounds for removal," Carl Tobias, a professor at the University of Richmond School of Law, told USA TODAY.

“I think there’s a rightful hesitation about removal from office, because obviously, you have to consider that to some degree you’re overturning an election,” Cox said.

Some Democrats eased up. State Del. Lamont Bagby, head of the Legislative Black Caucus, has called for Northam to resign, but he said he would let Northam "breathe a little bit, give him space to make the right decision."

More:In Northam controversy, Virginia remains haunted by its Confederate past

Northam won a rare call of support Tuesday from former U.S. Senator Joe Lieberman, the Democratic nominee for vice president in 2000.

"I think there's a rush to judgment that is unfair to him," Lieberman told CNN. "He ought to be judged in the context of his whole life."

Northam was bathed in the national spotlight Friday when a photo from his page in his 1984 medical school yearbook was published by the conservative website Big League Politics and went viral. The photo showed one person in blackface and another in a Ku Klux Klan robe.

More:Northam photo scandal and race relations

The governor initially apologized for being in the yearbook photo, but the next day, he reversed course, saying he doesn't believe he is in the photo and calling it "disgusting, offensive, racist." He did admit to wearing blackface as part of a Michael Jackson costume he wore for a dance contest in Texas in the early 1980s.

"I did not understand the harmful legacy of an action like that," Northam acknowledged.

Northam already had been the subject of controversy over a bill that would loosen restrictions on late-term abortions. Patrick Howley, editor of Big League Politics, said he obtained the yearbook photo from someone who was angry over some of the governor's comments supporting the measure.

Northam, in a radio interview, described a hypothetical situation in which an infant who is severely deformed or unable to survive after birth could be left to die. That prompted accusations from prominent Republicans that Northam supports infanticide.

Contributing: The Associated Press

More:'Sick, deeply toxic' culture: Ralph Northam hails from a time of trouble