(CNN) Lt. Col. Alexander Vindman, a key witness in President Donald Trump's impeachment inquiry, has told colleagues he expects to leave the White House's National Security Council in the coming weeks to return to the Defense Department, a source tells CNN.

Vindman said his departure from the council could come as soon as this month, the source said. That would be well ahead of the scheduled end of his time at the White House, which was originally slated to last until July. Vindman was appointed to his post in July 2018 and was asked to stay for two years.

The source did not say whether Vindman's departure was voluntary or at the behest of the White House, though aides inside the West Wing had speculated it could come soon as the impeachment trial was over.

Trump on Friday would not directly answer whether he wanted Vindman out of the White House, only telling reporters on the South Lawn, "Well, I'm not happy with him."

"You think I'm supposed to be happy with him? I'm not," Trump said, adding, "They'll make that decision. You'll be hearing. They'll make a decision."

Read More