Defense Secretary Jim Mattis brushed off reports that he could be leaving soon as “rumor,” joking that he is thinking about buying a house in the area.

The New York Times published a piece over the weekend alleging that Mattis is on the outs with his boss and could be leaving after the midterm elections, according to anonymous White House, congressional, and current and former defense officials.

Mattis dismissed those reports with a smile, telling reporters on Tuesday, “Those have been pretty good ones, haven’t they?”

“Of course I don’t think about leaving,” he joked. “I love it here. I’m thinking about retiring right here. I’ll get a little place down on the Potomac.”

The report came after anonymous assertions in journalist Bob Woodward’s new book that claimed that Mattis had disparaged President Trump. Mattis called those claims “fiction” and a product of someone’s “rich imagination” in a statement.

Trump later tweeted a Breitbart News story on Mattis’s statement, and batted down suggestions he was unhappy with him.

“He’ll stay right there. We’re very — we’re very happy with him. We’re having a lot of victories. We’re having victories that people don’t even know about. And he’s highly respected all over the world,” Trump said in response to a reporter’s question.

Mattis told reporters Tuesday he is focused on his job, and not to take rumors of his departure seriously.

“I’m thinking about doing my job each day,” he said. “I wouldn’t take it seriously at all.

“It’s like most of those kind of things in this town — somebody cooks up a headline, they then call to a normally chatty class of people, they find a couple of other things to put in, they add the rumor, somebody on the other coast starts writing the same thing – the next thing you know, you’ve got a story,” he said.

“It’ll die down just like – how many times have we been through this now, just since I’ve been here? It will die down soon, and the people who started the rumor will be allowed to write the next rumor too. Just the way the town is. Keep a sense of humor about it,” he added.