(CNN) In an extraordinary moment, Defense Secretary James Mattis was asked at the Pentagon this week if he had ever thought about resigning his post in the Trump administration.

The question came during one of Mattis' now regular off-camera, but on-the-record impromptu meetings with Pentagon reporters in the press area.

During a back-and-forth about whether Mattis is at odds with President Donald Trump on key issues, Mattis was asked: "It seems every time there's something the President says that upsets half the country, there's a call for the generals -- you, Kelly, McMaster -- should resign for something."

Then the crux of the question: "Have you ever had those thoughts cross your mind, in this administration, would that have crossed your mind?"

Mattis seemed to say he's never thought of resigning: "I don't care if it's Republican or Democrat, we all have an obligation to serve. That's all there is to it."

While he acknowledged early differences of opinion with the President on issues like US support for NATO and torture of terror suspects, Mattis stopped well short of addressing questions on whether he has any current differences with the President.

One former Defense Department senior official who worked closely with Mattis said the secretary has been frustrated with the White House.

The arrival of John Kelly as White House chief of staff may have brought some order to the Oval Office, several defense officials tell CNN. But it has not changed one fundamental way in which Mattis operates. Kelly, officials say, has continued to advise both Mattis and Secretary of State Rex Tillerson to keep low public profiles. Mattis already had been doing just that. He rarely appears on camera at the Pentagon and offers brief sessions with the media when traveling overseas.

He has not said publicly said so, but several defense officials have said he is concerned that if Trump -- an avid watcher of television news -- sees him on air too often, he will get angry, and Mattis worries he would lose influence with the President.

Instead he has turned to impromptu sessions with Pentagon reporters where he believes the absence of cameras helps him speak more candidly. Mattis insists he is not at odds with the media: "I don't see you all as adversaries. I see you as at times allowed to be more skeptical than I can be in a leadership role, and skepticism is part of a healthy -- keeping the organization healthy."

But he made clear he is reticent to appear on television because "people are trying to calculate each word."

There clearly are frustrations as well as differences with the White House. Mattis had promised Sen. John McCain, chairman of the Senate Armed Services Committee, a plan for the way ahead in the war in Afghanistan, but it was delayed for weeks because Mattis could not get the President to make key decisions on additional troops and strategy. Even now, after Mattis promised more public transparency on how many troops are in the war zone, he will not yet say how many additional forces he is sending.

One of the most stark examples of military leaders staking out their own views came in the wake of the violence in Charlottesville and criticism of Trump's remarks. All the members of the Joint Chiefs of Staff posted messages on Twitter condemning racism, and Mattis weighed in after he was publicly asked about it.

But it is on the issue of military versus diplomatic options for dealing with North Korea that the national security divisions may be most significant. Even after Trump's 'fire and fury" remarks, Mattis continued to emphasize that diplomacy comes first. This week Trump tweeted "talking is not the answer," when it comes to Pyongyang's weapons programs. After that tweet, which Mattis said he had read, he was asked at the Pentagon if the US is "out of diplomatic solutions." Mattis answered: "We're never out of diplomatic solutions."

Mattis insisted there was no daylight between him and the President.

"I agree with the President we should not be talking right now to a nation that's firing missiles over the top of Japan, an ally."

Overall, Mattis said: "I'll do my best to call it like I see it, but right now, if I say six and the president says half a dozen, they're going to say I disagree with him."