Washington(CNN) President Donald Trump on Wednesday said "we'll see" when asked if he would fire Health and Human Services Secretary Tom Price in the wake of reports he has used taxpayer-funded private planes for government business.

The President said he is "not happy" about the practice several of his Cabinet members have engaged in.

"I was looking into it and I will look into it. I will tell you personally, I'm not happy about it," Trump told reporters at the White House. "I am not happy about it I'm going to look at it. I let him know it."

However, a White House official said Wednesday that President Trump has "no immediate plans" to remove him from the Cabinet.

The President and Price talked Wednesday morning, aides said, during a brief call.

Price has apologized to the President for the unnecessary and repeated days of distraction, an official said, but did not offer to resign.

The official says this is different than Attorney General Jeff Sessions' situation because Trump believes Price has made a public mistake and made the administration look foolish.

"As it relates to this issue, as the secretary said over the weekend, he's heard the criticism and the concerns. He takes that very seriously and has taken it to heart," HHS press secretary Caitlin Oakley said in a statement. "The IG is going through a review and an audit of this process which we fully support."

Oakley also said Price "has initiated an internal departmental review of the procedures and processes that we go through for official travel to determine whether there are any changes or reforms that are necessary," and added that Price remains focused on his job.

"At the same time, the secretary and the department continues to work day and night to protect and enhance the health and well-being of the American people -- for example, providing assistance and medical care to Americans affected by the recent hurricanes and partnering with state and local community leaders to combat the opioid epidemic in America," Oakley said.

The issue came to the forefront after reports revealed that several of the President's Cabinet secretaries, notably Price, chartered private planes on the federal dime.

The Department of Health and Human Services' inspector general is investigating Price's decision to opt for private charter planes instead of commercially available means of travel.

Politico first reported Price's use of charter jets for official business earlier this month, and followed that article with another that said the secretary has traveled on charter flights at least 24 times since May, citing people familiar with his travel plans and a review of HHS documents.

Charter plane operators estimated 24 flights would have cost $300,000, Politico reported. Commercial trips would have cost thousands of dollars less.

Asked Wednesday whether he has confidence in Price, Trump declined to offer a clear answer, saying only that he is "looking into it."

And would he fire Price?

"We'll see," Trump offered.

The President's blunt words underscored his strong displeasure toward Price that could ultimately lead to his dismissal. Yet one administration official noted that Trump has also been furious at Attorney General Jeff Sessions and several other top advisers who he has not fired.

'Not good optics'

Earlier Wednesday, a top White House official said the administration was awaiting the internal and congressional inquiry into Price's use of private jets but declined to say whether the President had confidence in the secretary.

"I'm not going to comment on that because the President hasn't commented publicly," Kellyanne Conway, a senior White House adviser, told reporters.

Another White House official said the repeated reports of private jet use by Price were "not good optics," but said the President was focused on the Alabama Senate race and his tax speech Wednesday and had not talked extensively to aides about Price.

Inside the department, the mood is souring.

According to a former Obama administration official who is in close touch multiple Health and Human Services officials in the building, there has been growing frustration with each new report of Price's private charter plane use.

In particular, career officials are "aghast" and one person said they were "livid," according to the former Obama administration official.

Two current officials this source spoke to simply could not "believe he was using taxpayer money for a private plane."

Hill reaction

On Capitol Hill, the House Oversight Committee released a letter Wednesday revealing it was also investigating the use of private aircraft for official administration business.

In a letter addressed to Price, both the committee's top Republican, Rep. Trey Gowdy, and Democrat, Rep. Elijah Cummings, said they were "examining the extent to which non-career officials at federal departments and agencies either use government-owned aircraft for personal travel or private non-commercial aircraft for official travel."

The committee is looking at all Cabinet member travel, not just Price's.

Republicans on Capitol Hill have been slow to admonish Price, a former colleague, saying instead they are still trying to understand exactly what occurred.

One Senior Republican aide, speaking before Trump's latest remarks, pushed back on the idea that Price would face any serious consequences for his use of private planes saying unless an email revealed Price asked deliberately not to fly "with the masses," the controversy would be a few days story.

Republican Sen. Ron Johnson, chairman Homeland Security Committee said he wanted to "wait for the information" from the HHS inspector general before commenting.

Asked if he has confidence in Price, Johnson went to speak, then stopped himself and walked away.

Democrats, meanwhile, are more forthcoming with their outrage. Price, after all, maintained a reputation on Capitol Hill of being a fiscal hardliner and constant voice admonishing wasteful government spending.

"I'd like to introduce Secretary Price to the train," Sen. Tom Carper, a Delaware Democrat, quipped. "Not everybody can lease a plane like that and fly around the country. And I think as a public servant, you have to try to set some sort of example. "