Washington (CNN) President Donald Trump has grown irritated with his embattled Veterans Affairs secretary, David Shulkin, and is now making plans to remove him from his post, two sources familiar with the move tell CNN.

Trump has eyed Energy Secretary Rick Perry as a possible replacement, a person familiar with the matter said.

Trump and Perry discussed the possible move during a lunch at the White House Monday.

Shulkin's standing in the administration has been shaky for weeks as his agency devolves into turmoil, and though White House press secretary Sarah Sanders maintained just last week that he has done "a great job," Trump now wants him out.

A Shulkin adviser referred questions about this report to the White House. The White House declined to comment.

An Energy Department spokeswoman, asked about reports Perry is being considered to replace Shulkin, said: "I have no comment at this time."

Trump and his senior aides are frustrated with Shulkin because they believe he has undermined the White House on several occasions and is unwilling to work with other members of his agency who were appointed by the Trump administration, a source familiar with the matter tells CNN. His status as the sole Obama holdover in Trump's administration initially did not raise any issues, people familiar with the situation said. But the events of recent weeks made clear to Trump and his associates the importance of naming their own person to the job.

The prospect of Shulkin's ouster comes after months of infighting between Shulkin and Trump administration appointees at the department. At the root of the dispute is how far to go in privatizing the VA health care system. While early in his tenure Shulkin gained bipartisan acclaim for cleaning up a blighted agency, he began to butt heads more and more with political appointees over moving care outside the department, which currently serves more than 9 million veterans.

In recent weeks, Shulkin has put distance between himself and these political appointees, including those who run the agency's press operations, and has been essentially running a one-man crisis communications effort on his own behalf, telling multiple news organizations, including CNN, about efforts to oust him.

Shulkin met with Trump's chief of staff, John Kelly, at the White House on March 5 to discuss the agency's turmoil. After their meeting, Shulkin told The New York Times that he had the full confidence of the White House to fire staffers he felt were disloyal to him. But officials in the West Wing say that wasn't an accurate way to describe his meeting with Kelly and that afterward Kelly felt Shulkin had taken advantage of their conversation. He was frustrated that Shulkin went straight to the press to discuss what was said in their meeting.

Veterans groups react

Staff from multiple veterans' groups said Tuesday that they still support Shulkin's leadership of the department despite the prospect that Perry could replace him. These sources say they took Kelly at his word in a meeting last month when the chief of staff told them the White House supported Shulkin but that things have soured rapidly in recent days, with the secretary's standing seeming tenuous.

One source, who spoke with Shulkin over the weekend, said the secretary expressed to him that while things were a bit tense, he still felt that he had the support of the White House and was committed to his job.

The liberal advocacy group VoteVets has been one of a few to weigh in publicly, saying it will oppose Perry if he is named to the job, citing comments he made in a Newsmax interview in 2015 in which he said he supports moving more veterans' care to the private sector, an aim of conservatives that is not supported by most of the major veterans service organizations or by Shulkin.

In an interview last week, Will Fischer, VoteVets' director of government relations, said the goal of conservatives in the White House looking to oust Shulkin was to replace him with someone who, days after their confirmation, "will go about bulldozing VA facilities."