"It's clear they need to be my top priority right now, and this is not the time to start a new job as demanding as White House communications director," said Miller, whose wife is expecting their second child.