President Trump is quietly angling to have his daughter, Ivanka Trump, and her husband, Jared Kushner, removed from their roles at the White House, according to a report Thursday.

Trump has grown frustrated with Kushner as he deals with lingering issues related to his business ties, sources told the New York Times.

Although Trump has advised his daughter and son-in-law to keep their administration positions, he has privately asked White House chief of staff John Kelly for assistance in ousting them from the White House.

Kushner has come under fire this week after a report was published that his family business took million of dollars in loans from two companies that participated in White House meetings. That came after it was revealed last week that Kushner will not be able to access top secret intelligence because his White House security clearance was downgraded from the "Top Secret/SCI" level to "Secret" level.

Meanwhile, Ivanka Trump has irritated other staffers in the West Wing, including Kelly, for taking advantage of her position as first daughter, CNN reported recently.

In the past several weeks, a number of top aides have either left or signaled they will soon depart the White House, including White House communications director Hope Hicks.