John Kelly did not appear to be the married advisers' biggest fan

Former Chief of Staff John Kelly Says Jared Kushner and Ivanka Trump 'Have to Be Dealt With'

John Kelly, the former White House chief of staff, has revealed what it was like to work with Donald Trump‘s family members who currently serve as the president’s advisors.

While he did not mention Ivanka Trump and her husband Jared Kushner by name, Kelly, 68, said in an interview with Bloomberg Television’s The David Rubenstein Show that the two married staffers were “an influence” that had to be handled during his time in the Trump administration.

Get push notifications with news, features and more.

“They were an influence that has to be dealt with,” Kelly replied after being asked if it was complicated to have the president’s family working at the White House during his tenure as chief of staff.

Kelly immediately qualified his statement, adding that he was not referring to First Lady Melania Trump with his remarks.

“By no means do I mean Mrs. Trump,” he said. “The first lady’s a wonderful person.”

The White House did not immediately respond to PEOPLE’s request for comment.

Apparently, at one point while Kelly was chief of staff, Trump reportedly told him to fire Kushner and Ivanka.

According to Vicky Ward’s book Kushner Inc., which was based on 220 interviews, Trump told his chief of staff, “Get rid of my kids; get them back to New York.”

Image zoom John Kelly Bill Clark/CQ Roll Call/Getty

Trump’s command matches a report by the Washington Post in March 2018.

“[The president] has mused to Kelly that he thinks Ivanka and her husband should perhaps return to New York, where they would be protected from the blood sport of Washington and less of a target for negative media attention, White House officials said,” the Washington Post‘s Ashley Parker and Philip Rucker wrote.

However, in response to Ward’s book, the White House denied her written account of Trump’s reported conversation with Kelly.

“It’s sad, but not surprising, the media would spend time promoting a book based on shady anonymous sources and false information instead of all the incredible work Jared and Ivanka are doing for the country,” Press Secretary Sarah Sanders said in a statement to PEOPLE.

RELATED: Jon Voight Will Be a President Trump Appointee for the Next 6 Years … Here’s What That Really Means

Peter Mirijanian, who spoke on behalf of Kushner attorney Abbe Lowell, also denounced Ward’s account in a statement to the New York Times, which initially preview Ward’s manuscript in March.