One of Donald Trump’s senior communications staffers says he will not take the job of White House communications director after all.

Jason Miller said in a statement Saturday that he’s decided not to accept the role offered to him by Trump so he can spend more time with his family.

The resignation came amid reports that Miller had had an extramarital affair, and at the same time that Trump surrogate A.J. Delgado unleashed a bizarre series of tweets apparently aimed at Miller.

Politico reported that the affair has been the source of “turmoil” within Team Trump in recent days:

After POLITICO reporters received an anonymous email about the alleged affair, Delgado disclosed details of the relationship to senior officials in emails Thursday, the people close to the transition said. The nature of their relationship had been known to people involved in the Trump campaign and transition for “a number of months,” one source said.

Miller was one of the main spokesmen for Trump’s transition team. He says he and his wife are expecting their second daughter in January and “this is not the right time to start a new job.”

He says that after spending the past week with his family, “it is clear they need to be my top priority right now.”

Miller says he has handed all of his responsibilities to incoming White House press secretary Sean Spicer, who will serve as communications director as well as press secretary.