(CNN) Campaigning for the White House, Donald Trump repeatedly pledged to improve veterans' care. But President Donald Trump's choice of White House physician Ronny Jackson to serve as the next secretary of veterans affairs has been met with concern among Washington lawmakers and veterans' advocates who worry that Jackson lacks the experience to lead the government's second largest bureaucracy.

If Jackson is confirmed, he will inherit a Department of Veterans Affairs that has been faced with challenges that have spanned years. The department, which has 370,000 employees and serves more than 20 million veterans, has struggled to modernize with its inefficient health care system ensnared in crisis.

David Shulkin, fired by the President on Wednesday, departs at a critical moment with big decisions about the future of veterans' health care and its health records systems waiting in the wings.

Jackson has served for the last three administrations as a White House physician and is an active duty rear admiral in the Navy, but has little management experience.

That's led to a muted public response on Capitol Hill and private concerns that Trump's pick of Jackson, a man who has the President's affection and has been at the White House since 2006, is the right person for the job.

The top senators on the Senate Veterans Affairs Committee, chairman Johnny Isakson of Georgia, and ranking member Jon Tester of Montana, had little to say publicly about the President's choice. In separate statements, Isakson said he looks forward "to learning more about him," and Tester saying he wants to find out if Jackson is "up for the job."

Chris Lu, who served as the liaison to former President Barack Obama's Cabinet, told CNN, "VA is not only the largest health care system in the nation, but it also provides benefits for millions of veterans and their family members. Running this department is a massive management task. As accomplished a physician as Ronny Jackson might be, he brings less management experience to this job than any previous VA secretary."

A spokesperson for the former president declined to comment on Trump's decision to nominate Jackson.

Former Obama CIA Director John Brennan called Trump's pick of Jackson a "terribly misguided nomination" in a tweet Thursday.

"I personally know and greatly respect Ronny Jackson .... as a terrific doctor and Navy officer," Brennan wrote. "However, he has neither the experience nor the credentials to run the very large and complex VA. This is a terribly misguided nomination that will hurt both a good man and our veterans."

Jackson's views on some of the most pressing issues at the department remain unknown, particularly on the debate over allowing veterans more access to private doctors outside of the VA's health care system. Roughly one third of veterans already see private doctors, but the President and his allies are pushing for more private care, particularly the group Concerned Veterans for America, which is backed by the billionaire Koch brothers.

The group did not include a mention of Jackson in a press release after his nomination was announced. It said Shulkin had made "significant headway in reforming the department but ultimately became a distraction."

Asked how Jackson is qualified to oversee medical care for US veterans, a White House official told CNN Wednesday that Jackson would bring with him both a medical and military background to the department as well as past praise for his work in previous administrations.

Corey Lewandowski, Trump's former campaign manager, said on CNN's "New Day" that former VA administrators with more experience weren't getting the job done.

"I think if you look back and you look at the previous VA secretaries, those who supposedly had management experience who have come in, we have seen disasters under their regimes," Lewandowski.

Nevertheless, one of Shulkin's toughest critics on Capitol Hill says Jackson could face a bumpy ride.

Republican Rep. Mike Coffman of Colorado, one of the lawmakers who had called for Shulkin's removal, told CNN's "New Day" that to succeed at VA, Jackson would need to "clean house."

"This is an organization that's over half the size of the United States Army," Coffman said. "Unless he's going be tough, nothing's going to change."

Asked whether he believed Jackson needed management experience, he said, "it certainly helps," but added that Shulkin was an experienced manager and failed at the job.

"It's gonna be tough," he added. "I'll be frank with you, the odds are not with him."

Jackson's performance during an extended, at times testy, press conference over the President's health and cognitive fitness played a part in his nomination for secretary of Veterans Affairs, a White House official told CNN.

The official said Trump liked the way Jackson handled himself with reporters during the briefing in January.

Aside from his experience, it's the fight over privatization that may define Jackson's potential term.

In a fiery New York Times op-ed, Shulkin pointed to the policy fight as factor in his ouster, saying his successes at the agency "intensified the ambitions of people who want to put VA health care in the hands of the private sector."

"The private sector, already struggling to provide adequate access to care in many communities, is ill-prepared to handle the number and complexity of patients that would come from closing or downsizing VA hospitals and clinics, particularly when it involves the mental health needs of people scarred by the horrors of war," Shulkin wrote.

The leaders of veterans' groups sounded alarm on the pick, and said they weren't consulted with the White House. Many of them, in fact, had been pushing the White House to allow Shulkin to stay in the job, even as the President's attitude on Shulkin soured.

"I have no idea what to expect from him, quite frankly," Vietnam Veterans of America President John Rowan told CNN, adding that he was disappointed that the President had removed Shulkin, but he also expressed some relief at the pick -- it could have been worse.

"We were hoping that we prevailed, but at least he isn't appointing one of those rabid privatization people, so that's a plus," he said.

At AMVETS, executive director Joe Chenelly said that Trump's decision to oust Shulkin and replace him with Jackson had brought efforts to reform and modernize VA "to a screeching halt."

"We don't see anything in his bio that makes him fit to lead," Chenelly told CNN, adding that his group is "a long way to supporting the nominee."

Veterans of Foreign Wars echoed that point in a statement Thursday, saying Jackson's background does not show any experience working with the Department of Veterans Affairs or with veterans or managing any organization of the size of the agency, "so the VFW will be closely monitoring his Senate confirmation process."

The group also praised Shulkin for his helming of a department rocked by a crisis the year before he took over, saying that under his tenure, the agency saw successes ranging from "increased accountability and transparency to a beefed-up education package and increases in overall funding and patient satisfaction."

Paul Rieckhoff, the chief executive of the Iraq and Afghanistan Veterans of America, told CNN's "New Day" that the move to replace Shulkin would only add to the instability that already plagues VA, noting that since he returned from Iraq in 2004, three VA secretaries have either resigned in scandal or been fired.

"The VA's a monster for any leader to try to manage effectively. And when the changeover happens, it's very disruptive to the front line personnel at the VA, it's very disruptive to the veterans that depend on it for care," he said.

Of Jackson, Rieckhoff added, "he's going to have a very hard row to hoe to try to convince people that he's qualified."