Former Veterans Affairs Secretary David Shulkin told Fox News' "Special Report" Thursday night that President Trump "wasn't happy with the pace" of reforms that the beleaguered department.

"I came to Washington from the private sector for one reason: to fix the VA," Shulkin told host Bret Baier. "I know this matters a lot to the president and we as a country have a long way to go to get the VA in proper shape."

Trump fired Shulkin, the lone Obama administration holdover in his Cabinet, late Wednesday amid a mounting internal rebellion at the VA and an ongoing ethics controversy. At a speech in Ohio earlier Thursday, Trump did not mention Shulkin by name, but did say that "I wasn't happy with the speed with which our veterans were taken care of ... We made changes because we want them to have choice."

Shulkin wrote in a New York Times op-ed Thursday that his removal had been engineered by people who "saw me as an obstacle to privatization."

"[T]here are a number of political appointees within VA that are pushing to privatize much faster than I think is in the interest of veterans and is the safe way to care for our veterans," Shulkin told "Special Report," "and I’ve stood up [to] them because I was in Washington and I was working for the president on behalf of our veterans."

However, Shulkin admitted, "I don’t think the president should be happy with where we were."

"This is a system that needs to be pushed. Many of the bureaucratic elements need to be restructured," he added. "I’ve started that process, I hope Dr. [Ronny] Jackson will continue that process because it needs to be done."

Shulkin also addressed a report by the VA's internal watchdog that concluded that the secretary had improperly accepted Wimbledon tennis tickets and that his then chief of staff had doctored emails to justify his wife traveling to Europe with him at taxpayers' expense.

"Everything that I did on this European trip was pre-approved by Ethics [and] was consistent with past policies at VA," said Shulkin, who claimed that the watchdog report was "was used in a political way to try to remove me."

The Associated Press contributed to this report.