Richmond, Virginia (CNN) Virginia Gov. Ralph Northam remained in power Monday but is having a difficult time finding allies, begging his Cabinet members to give him the chance to prove he was not the person pictured in a racist photo that surfaced Friday.

Northam oversaw a regularly scheduled Cabinet meeting Monday morning that a source inside the meeting described as "solemn."

According to that source, the governor specifically said that if he resigns, he would be resigning as a "racist for life," and that the only way he can clear his name is to stay in office and convince people that he is not in that photo and that the photo does not represent who he is.

Not one member of the Cabinet offered a resignation, nor did any threaten to resign, the source said. This person said there was tension in the room and that many expressed concern that, despite the governor's desire to clear his name, he would be unlikely to regain the confidence of Virginians.

The Cabinet members are collectively "struggling" over this situation and not one of them thinks he is a racist -- but his explanation about what happened 35 years ago is "untenable," said the source.

"We are struggling here, there is no better way to put it. We all love the governor," the source in the room told CNN. "We don't think he has a racist bone in his body -- but he can't seem to explain this, and it is making it very difficult for us to stick by him."

Northam said at a later all-staff meeting that he needs more time to decide his path forward after a decades-old racist photo surfaced on his yearbook page, a Virginia Democrat briefed by multiple officials who were in the meeting told CNN. The Democrat said the meeting was intense, with both Northam and his chief of staff described as "emotional."

A Virginia Democrat briefed on Monday's staff meeting told CNN that Clark Mercer, the governor's chief of staff,urged other senior aides to give the governor the time he and his wife were looking for. An administration official who was in both meetings told CNN that the decision was "completely up to us" and there was no "obligation to do anything."

"The chief of staff was very clear that as members or the staff that we all had a decision to make and that decision was completely up to us," said the official.

At this point, there is no sign of a rallying around the embattled governor, something that will need to happen if he hopes to meet his own personal standard of staying on only if he can continue to effectively govern. Monday morning, protesters gathered outside the governor's mansion in Richmond.

The Democratic governor hastily called a meeting of his top administration officials of color on Sunday night to see if anyone with deep ties to him would be willing to stand by him during a firestorm of controversy over a decades-old racist photo.

The meeting did not go Northam's way. Not one person during the gathering told the governor that he should stay and fight, a source with knowledge of the conversation said.

Lt. Gov. Justin Fairfax, who is black and would assume the governorship were Northam to step down, told reporters in Richmond Monday that he didn't have anything to add to his statement from over the weekend, in which he said he could not condone Northam's past actions but did not call on him to resign.

"I believe that the governor has to make the decision that is in the best interest of the commonwealth of Virginia," Fairfax said Monday. "I know that there are many others who have called on him to resign. I am in a unique position, obviously, as the lieutenant governor and someone who would have to assume that office in the event that he were to resign, so I have to be very circumspect and I have to think of the people or the commonwealth."

He added that the last time he spoke with Northam was "probably a couple of days ago."

Despite calls for Northam's resignation, GOP Speaker of the Virginia House of Delegates Kirk Cox told reporters Monday morning that there is no will within the General Assembly to forcibly remove Northam from office and that the current situation does not fall within the two constitutional provisions that allow for a governor to be forcibly removed.

After initially saying Friday night that he was in the photo that appeared on his page of his 1984 medical school yearbook which shows one person dressed in blackface and another in the KKK's signature white hood and robes, Northam is now insistent that he is not one of the two people pictured. He is hoping to convince those under his charge that he be given the opportunity to explain what happened and that the picture does not reflect the person he is.

The governor had hoped a news conference on Saturday afternoon would turn the tide in his favor. He attempted to make the case that he was not the person in the photo, but did admit to appearing in blackface at another time that same year, at a dance competition in San Antonio.

The fallout from the news conference was swift and damning. Many of the prominent leaders who had already called for his resignation renewed that call. The few that had held back then called for him to step down, including the state's two US senators, Mark Warner and Tim Kaine, who had been privately encouraging him to resign.

This story has been updated.

CLARIFICATION: This story has been updated with additional reporting to clarify the characterization of Clark Mercer's comments to staff during Monday's meeting.

CORRECTION: This story has been updated to accurately reflect that the racist photo which shows one person dressed in blackface and another in the KKK's signature white hood and robe was published in the medical school yearbook 35 years ago.