Some of President Donald Trump's allies fear the departure from the White House of a man many describe as an emotional anchor, according to Bloomberg.

Keith Schiller is one of the president's closest confidants and has worked for Trump for nearly two decades, the report says. Schiller is director of Oval Office operations, but many surveyed say he acts as a protector and wing man to the president.

Two people close to Schiller told Bloomberg he never planned to stay in his White House position for very long, due to long hours and low pay. He plans to return to the private security industry, where his annual salary should greatly exceed the $165,000 he currently receives from the government, according to the report.

Schiller declined to comment, Bloomberg said. The White House did not immediately respond to CNBC's request for comment.

And Schiller's easy access to the president is increasingly restricted after Trump appointed former Gen. John Kelly as chief of staff. With doors around the White House now closed, both figuratively and literally, Schiller has told friends he doesn't like working under Kelly as much, according to Bloomberg.

Other people close to Trump worry, however, that Schiller's departure may result in an unbalanced presidency due to the close-knit nature of their relationship, Bloomberg says.

Schiller began working for Trump as a part-time body guard in 1999, according to the report. Since Schiller's ascension to the Trump Organization's head of security in 2004, he has been closely involved in relationships with the company's employees. During the campaign, Schiller sat in on almost all phone calls and conversations the then candidate Trump had.

Trump will also lose his go-to political executioner in Schiller. According to Bloomberg, Schiller told former FBI director James Comey that Trump was firing him. And when Trump was angry with the preparations for a recent rally in Phoenix, Schiller told White House aide and planner George Gigicos that the president did not want Gigicos to organize any other similar events.

Read the full Bloomberg report here.