US President Donald Trump speaks as Defense Secretary James Mattis (L) looks on during a meeting with senior military leaders in the Cabinet Room of the White House on October 5, 2017.

Former Defense Secretary James Mattis, who has declined numerous opportunities to criticize President Donald Trump's policies since leaving his Cabinet post, said Tuesday he may not stay silent forever.

In a conversation with Council on Foreign Relations President Richard Haass, Mattis said that "when the time's right to speak out about policy or strategy, I'll speak out."

Mattis stepped down in December. His letter of resignation cited his disagreements on policy grounds with the president, explaining his "core belief" that "our strength as a nation is inextricably linked to the strength of our unique and comprehensive system of alliances and partnerships."

Trump has "a right to have a Secretary of Defense whose views are better aligned" with his own, Mattis wrote.

On Tuesday, Mattis unpacked his commitment to keeping mum on the Trump administration.

"I don't believe that if you leave an administration over a matter of policy — and I made that clear in the letter — that you then get out and start talking," Mattis said, "and what we commonly call a 'kiss-and-tell' now, I don't think that's the right thing to do."

He added that he didn't want to make things "more difficult" for officials still in the administration by commenting from "the cheap seats, not responsible for anything."

And he pushed back on the the president's detractors, who have pressured Mattis to speak out against Trump as a matter of responsibility or principle. "I've led a responsible life. I know what responsibility is," Mattis said.

He continued: "It's not the right place for the defense establishment to be dictating or somehow influencing, I think, the political associations, the political qualifications of people."

"I've got a lot of trust in the American people. I don't lose sleep over the American people," he said.

The general's new book, "Call Sign Chaos: Learning to Lead," released Tuesday, also does not include any attacks on the president. Mattis said the book had been in the works years before he joined Trump's Cabinet.

"I've frustrated everyone so far," Mattis joked at one point.

In a clip from an interview with CNN, to be aired Tuesday evening, Mattis made a similar argument.

"They have big responsibilities right now," Mattis said, "and I don't believe that I add anything to it by representing contrary views or something like that."

But "there'll come a time when it's right for me to talk about strategy and policy," he added. Asked when that might be, Mattis said, "I will know it when I see it."

When pressed to say if that time might come before the 2020 presidential election, Mattis said, "I can't say that."

Before he retired as a four-star general in 2013, Mattis had led U.S. Central Command, which directs military operations and oversaw the wars in Iraq and Afghanistan.