Veterans Affairs Department Secretary David Shulkin has fallen from grace, but he's still trying to claw his way back up.

Shulkin — the only holdover from the Obama administration to serve in President Donald Trump's Cabinet — told lawmakers at a budget hearing Thursday that he was battling "politics and distractions" so that he could "focus on vets."

"Let's discuss the proverbial elephant in the room — some reports even mentioned you have an armed guard stationed outside your office," noted Rep. Charlie Dent, R-Pa., during Shulkin's appearance before the House Appropriations Committee.

"Is all of this squabbling affecting your mission to serve veterans?" Dent asked.

Shulkin quickly pivoted, saying he would "prefer not to discuss the specifics" of his security detail, which reportedly has been posted there as the secretary battles political appointees within his own department.

He said he was speaking at the committee "for one reason, and that is to improve the lives of veterans."

"A lot of people are more interested in politics," Shulkin said. "I'm interested in getting the job done. I do believe we are getting back on track. And there is a lot of work to do."

He might not get much chance to do that. Shulkin's job of overseeing the nation's biggest integrated health-care system was widely considered to be hanging by a thread Friday, along with that of other high-level officials in the Trump administration.

The White House and the VA did not immediately respond to a request for comment from CNBC.

Shulkin's now-bruised reputation and relationship with Trump — who previously had been a big fan of his — stem from several issues.

One is the outcry over an internal watchdog's finding that Shulkin used taxpayer money for his wife's airfare during travel with him on an official trip to Europe last year.

Another problem is the long-standing criticism of the quality of health care delivered to veterans, waste within the VA and Shulkin's resistance to increasingly outsource vets' health care to private providers as many conservatives want to see happen.