Washington (CNN) White House chief of staff John Kelly is facing stiff blowback as he contends with another embarrassing staffing scandal , though there's no indication his own job is at immediate risk, according to two sources who spoke to President Donald Trump on Thursday.

There is growing frustration among Trump's aides and allies in and out of the White House that Kelly badly mishandled the abuse claims made against staff secretary Rob Porter, one of his top deputies. Sources say he knew for months about some claims that Porter physically and emotionally battered two ex-wives, yet didn't conduct an internal investigation into their veracity.

The unfolding scandal is raising serious questions about whether Kelly is calming -- or simply contributing to -- the ongoing chaos of Trump's West Wing . Despite the swirl of controversy, Kelly's job seems secure for now, in no small part because few close to the President can readily identify a natural replacement for Kelly should he go.

"There are a lot of knives out for him," one official acknowledged, "but the President needs and trusts John Kelly."

"I think it's fair to say we all could have done better dealing with this over the last few days," he said.

Shah declined to say when Kelly was first aware of the allegations about Porter, saying only that "he became fully aware of these allegations yesterday."

"I am not going to get into the specifics of what may have emerged from the investigation," Shah said.

Background check

JUST WATCHED Porter's ex-wife says he is abusive and angry Replay More Videos ... MUST WATCH Porter's ex-wife says he is abusive and angry 02:49

One of Porter's ex-wives, Colbie Holderness, confirmed to CNN that she told the FBI during an interview for Porter's background check that Porter punched her.

In previous administrations, problems revealed during an FBI background check would have immediately gone to the White House counsel's office, who would then have conducted a review, according to a lawyer who worked in the Obama White House.

The White House has declined to say when specific members of the administration, including White House counsel Don McGahn, became aware of details. But current and former administration officials said Kelly, McGahn and deputy chief of staff Joe Hagin were all aware of some of the claims by the fall, according to officials.

Kelly's defenders in the West Wing insist that Porter misled him and others about the domestic abuse allegations, despite the outlines of the charges being known. When Porter became aware the allegations would soon become public, he implored those he was closest to in the administration -- including Kelly -- to defend him, people familiar with the situation said.

For Kelly, those denials were enough to warrant the effusive statement of support that was released Tuesday evening, which described Porter as "a man of true integrity and honor." He stuck by that statement on Wednesday, even as photographs of one of the ex-wife's bruised face emerged. By the evening, however, it became clear his position was untenable.

"I was shocked by the new allegations released today against Rob Porter. There is no place for domestic violence in our society," he wrote in a statement released after 9 p.m., where he remained unwavering in his support of Porter.

"I stand by my previous comments of the Rob Porter that I have come to know since becoming chief of staff, and believe every individual deserves the right to defend their reputation," he wrote. "I accepted his resignation earlier today, and will ensure a swift and orderly transition."

By Thursday night, Kelly sent a lengthy memo to White House staffers addressing the allegations made against Porter.

"While we are all processing the shocking and troubling allegations made against a former White House staffer, I want you to know that we all take matters of domestic violence seriously," Kelly wrote. "Domestic violence is abhorrent and has no place in our society."

Questions mounted

JUST WATCHED Senator: Shame on WH if covering up abuse claims Replay More Videos ... MUST WATCH Senator: Shame on WH if covering up abuse claims 01:16

Questions mounted through the day about why Kelly didn't take action sooner to dismiss Porter after the abuse allegations emerged. He was up early to accompany Trump to the National Prayer Breakfast, held inside a hotel ballroom in Washington.

He was smiling as he stepped from the White House South Portico with Trump and two other aides, senior counselor Kellyanne Conway and policy adviser Stephen Miller. He waved the two other advisers into the presidential limousine while he himself rode to the breakfast in a staff vehicle.

One official who spoke with Trump and Kelly on Thursday denied the President was enraged at the botched initial reaction to the Porter allegations.

"He's not mad at John Kelly," the official said of the President. "He's mad and disappointed at Rob Porter."

Trump is not known for opening his circle to newcomers, but he was taken by Kelly's military pedigree and asked him in late 2016 to join the administration as secretary of homeland security. In a speech last year, Kelly acknowledged he "literally did not know Mr. Trump at all" or "know anybody that knew Mr. Trump."

Trump later announced that Kelly would replace his first chief of staff, Reince Priebus -- before he'd even told Kelly of his decision. His mandate was to rein in a warring West Wing and streamline the President's decision-making.

But the rigorous system that Kelly enacted when he entered his job -- which was partly enforced by Porter -- has vexed both staffers and the President himself, who was accustomed to a free-wheeling system of access. In recent weeks, aides have increasingly protested the restrictions Kelly has placed on their access to the President. A frequent complaint is how Kelly refuses their requests to meet with Trump, preferring that all information that reaches the Oval Office is first seen by him.

One White House official says that while staffers -- including Ivanka Trump, the President's daughter and senior adviser -- "have their issues" with Kelly, they realize he is critical to stabilizing the chaotic West Wing.

"She won't do anything to undermine him," one official says, adding: "If Kelly leaves, the rest of the ship goes with him."

As for Kelly's future at the White House, another source familiar with their relationship says that while there are certainly days the President is frustrated with him, he is not necessarily at the point that he wants him gone yet.

Kelly, too, has aired frustrations at the job -- including threatening to quit in past disputes with the President, according to people familiar with the episodes. But he's told allies that he feels a patriotic duty to see Trump's agenda enacted.

"I get the sense that he feels like he's more on edge right now in terms of the things he's saying and I think that probably reflects a lot of frustration," said Leon Panetta, who as secretary of defense worked closely with Kelly, who was his senior military aide. Panetta also served as chief of staff to President Bill Clinton. "It probably is the toughest job he's ever had. In the end he is still a human being."

"You face the same challenges every day," Panetta said. "It starts to impact both on your viewpoints and just your ability to stick with it."

CORRECTION: This story has been updated to correct the spelling of the last name of White House counsel Don McGahn.