Washington (CNN) President Donald Trump has grown increasingly frustrated with Hope Hicks, his communications director and one of his closest confidantes, amid the fallout from the Rob Porter scandal, people familiar with the matter say.

Meanwhile, the President has told associates he's dismayed at how the allegations involving his former staff secretary accused of domestic abuse were handled, but he isn't certain how to solve the mushrooming controversy, a person familiar with the conversations tells CNN.

Trump was not consulted when Hicks and several other aides drafted a White House statement defending Porter, and he is under the impression that Hicks has let her romantic relationship with Porter cloud her judgment, a source familiar said.

In the aftermath, Trump has told associates he feels that Hicks put her own priorities ahead of his. However, there is little to indicate that Hicks' standing is in jeopardy.

Speaking during the White House briefing on Thursday, spokesman Raj Shah said Hicks had recused herself from some matters related to the Porter fallout. Porter was in the building for a short period to clean out his desk, Shah said.

Read More