President Trump has removed Anthony Scaramucci from his new job as communications director, on his 11th day on the job -- he was named to the position on July 21.

"Anthony Scaramucci will be leaving his role as White House Communications Director. Mr. Scaramucci felt it was best to give Chief of Staff John Kelly a clean slate and the ability to build his own team. We wish him all the best," White House press secretary Sarah Huckabee Sanders said in a statement.

Scaramucci was escorted out of the White House, a source told CBS News.

This comes the same day that retired Marine Corps Gen. John Kelly was sworn in as the new White House chief of staff. The New York Times first reported the development, which said that Kelly asked Mr. Trump to remove Scaramucci from his role. The report added that Kelly also made clear Monday morning that he's now in charge. Kelly wanted "more structure, less Game of Thrones," a source close to the White House told CBS News. The source added that Kelly will have far more latitude than Reince Priebus & did a good job negotiating terms of his service -- "either I'm a chief of staff in name, or I'm the chief of staff."

Get Breaking News Delivered to Your Inbox

Scaramucci had previously bragged that he would report directly to the president rather than the chief of staff.

Just hours earlier, the president tweeted that there is no chaos at the White House.

Highest Stock Market EVER, best economic numbers in years, unemployment lowest in 17 years, wages raising, border secure, S.C.: No WH chaos! — Donald J. Trump (@realDonaldTrump) July 31, 2017

As the news began circulating Monday afternoon, outgoing press secretary Sean Spicer came out of his office where he had huddled with various people. He joyously asked, looking at a dozen or so reporters in the hallway, if there was a surprise party.

At a Cabinet meeting earlier in the day, Mr. Trump said, "I predict that General Kelly will go down, in terms of the position of chief of staff, one of the great ever. And we're going to have a good time, but much more importantly, we're going to work hard and we're going to make America great again."

Mr. Trump named Kelly, who had been serving as Homeland Security secretary, to the new role on Friday, replacing Reince Priebus. Priebus said that he had resigned from the position, which he had served in since the inauguration in January.

Anthony Scaramucci, named Donald Trump's new White House communications director speaks during a press briefing at the White House in Washington, D.C. on July 21, 2017. Getty

Scaramucci was viewed as a provocative figure who bumped heads with both Priebus and Sean Spicer, who resigned from his job as press secretary the same day that Scaramucci was chosen to lead the White House communications operation. Sarah Huckabee Sanders has since taken over as press secretary.

Last week, he came under fire for comments he made during a phone call with New Yorker reporter Ryan Lizza. He said that he believed that Priebus had leaked information about the dinner he attended with the president that night.

"They'll all be fired by me," he said. "I fired one guy the other day. I have three to four people I'll fire tomorrow. I'll get to the person who leaked that to you. Reince Priebus -- if you want to leak something -- he'll be asked to resign very shortly...Reince is a f******* paranoid schizophrenic, a paranoiac."

Then using a vulgar colloquialism, Scaramucci said that Priebus had tried to block him for months, and was now giving former Fox News executive Bill Shine the same treatment. The entire exchange between Lizza and Scaramucci was detailed in a story published online Thursday by the New Yorker.

Scaramucci is the second communications director to leave the position. In May, Michael Dubke resigned from the gig, saying that he was leaving for personal reasons.

CBS News' Major Garrett, Jacqueline Alemany and Katiana Krawchenko contributed to this report.