Former Secretary of Veterans Affairs David Shulkin is speaking out for the first time about a "shadow government" he says he undermined him at every turn, and ultimately ousted him in April 2018.

In a new book, "It Shouldn't Be This Hard to Serve Your Country," Shulkin paints a troubling picture of a "toxic, chaotic and subversive" work environment.

Shulkin says that a group of political appointees within the VA went over his head to hold their own meetings with White House officials on policy issues, and ultimately succeeded in ousting him from his job.

Specifically, Shulkin blames those political appointees for leaking his travel schedule and lodging an allegation that he used government funds to pay for his wife's travel to vacation in Europe.

Visit Business Insider's homepage for more stories.

As the Trump administration faces allegations that it undermined career State Department officials amid the ongoing impeachment inquiry, a former cabinet official is speaking out about a "shadow government" he says he undermined him at every turn.

In a new book, "It Shouldn't Be This Hard to Serve Your Country," former Secretary of Veterans Affairs David Shulkin paints a troubling picture of a "toxic, chaotic and subversive" culture in President Donald Trump's administration, according to NPR.

Unlike many of Trump's other cabinet officials, Shulkin had extensive experience in government, previously serving as the Undersecretary for Health in the VA under President Barack Obama's administration. Shulkin holds a medical degree from the Medical College of Pennsylvania.

Read more: Ex-Ukraine envoy says she was fired on 'unfounded' and 'false' grounds after standing up to Trump and Giuliani

In the interview, Shulkin says that while the time after Trump's inauguration was very productive, "there were a couple of events that made me question and wonder whether there was a dual path of decision making in the White House."

Shulkin says he ultimately came to realize that "there were clearly two paths when it came to veterans issues: the one that came from me, and the ones that were coming from other people," naming the "other people" as political appointees within the VA that went over his head, and held their own meetings with White House officials.

In August of 2018, the investigative journalism outlet ProPublica reported that not just political appointees, but wealthy members of Trump's Mar-a-Lago club in Palm Beach, Florida, were involved in VA affairs and gave direction to VA employees on key policy issues despite not working in the government.

Shulkin said that the largest schism between himself and the political appointees and outside forces influencing veterans' policy was over the extent to which VA services should be taken over by the private sector. The VA is the country's largest integrated health care system, with an annual budget of $220 billion.

While those appointees pushed for vast privatization of VA, Shulkin argued for a more measured approach.

"I was in agreement that every veteran should get a choice about where they get their care, but I wanted to do this in a way that would not destroy the current VA system, because I believe the VA is doing things the private sector is not doing well," Shulkin explained, worrying that privatizing too quickly or widely would undermine the quality of care.

"That was the issue I fought hardest for, and the one that ultimately ended up costing me my job," Shulkin said, arguing that those political appointees ultimately determined Shulkin needed to be pushed out, and as he tells it, "perfected the art of the leak."

Specifically, Shulkin blames those political appointees for leaking his travel schedule and lodging an allegation — which Shulkin vehemently disputes — that he used government funds to pay for his wife's travel to vacation in Europe.

In reality, Shulkin says, he and his wife, who are both doctors, had both been invited to a Five Eyes security conference in Italy and their travel plans were approved by ethics officials. But the allegation that he misappropriated taxpayer funds was enough to cast a cloud over his job.

Then one day, Shulkin recalled, his deputy security brought him an email he found on a copy machine from one political appointee to other appointees outlining a plan to remove him, his deputy secretary, other senior officials, and "put in place people who agree with us."

Read more: Fired Veterans Affairs Secretary speaks out, says he didn't resign after the White House insists he did

Even when Shulkin tried to fire the political appointees trying to destroy his career, he says the White House stepped in and prevented them from being removed.

And even after assurances from Trump himself and former White House chief of staff John Kelly that his job was secure, Trump blindsided Shulkin in April 2018 by ultimately firing him in a tweet.

"I think it's a duty to share what one's learned in government," Shulkin said. "I am not trying to throw mud at anybody or make criticisms of any particular political party. I'm trying to be transparent about my experience so that we can learn from it...I am very concerned about the future of public service."