Washington (CNN) President Donald Trump has stayed in touch with Rob Porter, a former aide who resigned after accusations of spousal abuse surfaced -- and Trump has said he wishes he could have Porter back at the White House, The New York Times reported Monday, citing three people familiar with the conversations.

The Times said it was unclear how often the President has spoken with Porter since the former aide stepped down, but that the calls increased as more people close with Trump have left the White House recently.

The Times said also that Trump has told advisers he realizes he probably could not bring back the staffer who exited following the accounts of alleged abuse by his ex-wives.

Porter denied the allegations, and the White House offered kind words as Porter departed, setting off a wave of renewed scrutiny over White House staffers operating on interim security clearances.

Both the White House and Porter declined to comment to the Times.