Story highlights Scaramucci said he hasn't spoken with Trump about a return

He was fired after less than two weeks as communications director over the summer

(CNN) Former White House communications director Anthony Scaramucci on Thursday denied that he's been saying President Donald Trump wants him back in the West Wing.

"I don't even know why people are saying that," Scaramucci told CNN "New Day" host Chris Cuomo. "I never had a conversation with the President about that or his family. I'm more focused on my house, Chris, than the White House."

On Monday, the Daily Beast reported that Scaramucci told friends and associates that Trump and Ivanka Trump had spoken with him, asking him to return to the White House. Scaramucci pushed back against the report, tweeting it wasn't "happening anytime soon."

Scaramucci was sacked from the White House after less than two weeks on the job following an expletive-laden interview with The New Yorker.

"You've never heard any whining from me," Scaramucci said Thursday. "I don't say I resigned ... I got fired by Gen. (John) Kelly because the comments I made were picked up on a recorded line by somebody I trusted ... I owned it and got fired."