The departure of Keith Schiller, a longtime aide and confidante of President Donald Trump’s, has left the President’s allies worried how Trump will act without Schiller at his side, Bloomberg News reported Tuesday morning.

Schiller, who led the Trump Organization’s security operations before working in the White House as director of Oval Office operations, has decided to leave the White House, CNN and CBS News reported over the weekend. Schiller had never planned to stay long at the White House, and he will soon return to a job in private security, according to Bloomberg News.

His decision to leave the White House was accelerated by the new regime implemented by chief of staff John Kelly, who revoked Oval Office walk-in privileges from Schiller and several others, according to CBS and Bloomberg News.

As a longtime employee of Trump’s, Schiller was one of Trump’s most loyal and trusted aides. The President dispatched Schiller to fire FBI Director James Comey and former White House aide George Gigicos.

Trump was “crushed” by Schiller’s decision to leave the White House, per Bloomberg News. Several allies told Bloomberg news that Trump was losing an “emotional anchor” with Schiller’s departure, and that the loss could lead Trump to butt heads with Kelly.