(CNN) Rear Adm. Ronny Jackson has withdrawn as President Donald Trump's nominee to lead the Department of Veterans Affairs, he said in a statement Thursday morning.

His nomination was hampered by a flurry of allegations about Jackson's professional conduct.

In a statement announcing his withdrawal, Jackson slammed allegations of improper behavior leveled against him as "completely false and fabricated."

"If they had any merit, I would not have been selected, promoted and entrusted to serve in such a sensitive and important role as physician to three presidents over the past 12 years," Jackson said. "Going into this process, I expected tough questions about how to best care for our veterans, but I did not expect to have to dignify baseless and anonymous attacks on my character and integrity."

Jackson said he was motivated to withdraw from consideration because the allegations against him "have become a distraction" for Trump and his agenda.

"Unfortunately, because of how Washington works, these false allegations have become a distraction for this President and the important issue we must be addressing -- how we give the best care to our nation's heroes," Jackson said. "While I will forever be grateful for the trust and confidence President Trump has placed in me by giving me this opportunity, I am regretfully withdrawing my nomination to be Secretary for the Department of Veterans Affairs."

Speaking on "Fox & Friends" Thursday morning, Trump said Jackson's withdrawal was not unexpected.

"I even told him a day or two ago I saw where this was going," Trump said.

JUST WATCHED New allegations against Trump's VA pick Replay More Videos ... MUST WATCH New allegations against Trump's VA pick 02:54

Allegations mounted

On Wednesday evening, the release of a two-page document written by Democratic staff on the Senate Veterans Affairs Committee that included a list of allegations from the committee's conversations with nearly two dozen of Jackson's former and current colleagues made Jackson's chances at survival more uncertain.

According to the summary released by Senate Democrats -- the contents of which remain under investigation by lawmakers and have not yet been substantiated -- Jackson was allegedly "abusive" to his colleagues, loosely handled prescription pain medications and was periodically intoxicated, even once wrecking a government vehicle while drunk.

Jackson has not responded to the allegations in their totality, but told reporters on Wednesday that he did not wreck the government car and had indicated at the time that he planned to continue in the nominating process.

Republican and Democratic lawmakers on the Senate Veterans' Affairs Committee had raised concerns this week about allegations involving Jackson, the White House physician, and had started to review the allegations in an effort to determine whether they were sufficient to upend his nomination.

Jackson had been expected to appear on Capitol Hill on Wednesday for a confirmation hearing, but that hearing was postponed indefinitely as members were assessing allegations about the White House physician's behavior.

On Wednesday, before the allegations from the Democratic staff of the veterans' panel became public, the White House mounted a robust defense of Jackson, with White House press secretary Sarah Sanders telling reporters he had an "impeccable" record. Sanders suggested that Jackson's current position as the White House physician meant that he had been more thoroughly vetted than other Cabinet nominees.

At one point on Wednesday, Trump raised the prospect of going into the briefing room himself to stick up for Jackson, to say that he is a good guy and had his support. But several senior administration officials, including Sanders, advised against that move.

But later Wednesday, after the allegations included in that Democratic document were released, Trump himself began to wonder out loud whether Jackson should step aside "before things get worse" and White House aides were preparing for that possibility, officials told CNN.

JUST WATCHED White House: VA pick's record is impeccable Replay More Videos ... MUST WATCH White House: VA pick's record is impeccable 01:37

More upheaval for Trump's Cabinet

Trump had defended Jackson even after the allegations against him surfaced.

On Tuesday, the President called Jackson "one of the finest people I have ever met." Trump said he was not aware of the specifics of the allegations and added that it was up to Jackson to decide whether to press forward. "It's totally his decision," the President said, adding, "I don't want to put a man through a process like this. It's too ugly and disgusting."

But Trump also acknowledged that Jackson lacks a background in running a government agency.

"There's a lack of experience," he said.

JUST WATCHED Trump taps White House doctor as VA secretary Replay More Videos ... MUST WATCH Trump taps White House doctor as VA secretary 02:35

A surprise nominee

When Trump announced his intent to nominate Jackson in March, the decision came as a surprise on Capitol Hill, where members on both sides of the aisle raised questions about whether Jackson was qualified to lead the sprawling Department of Veterans Affairs. While Jackson served as White House physician under three presidents, his policy views were relatively unknown and some lawmakers expressed concern over whether he had the managerial experience necessary to steer the department.

A graduate of Texas A&M University, Jackson started active-duty naval service in 1995, according to his official US Navy biography . He was chosen to serve as White House physician in 2006 during the administration of George W. Bush, and later served as the physician for then-President Barack Obama.

In January, Jackson made headlines after declaring he had "no concerns" about Trump's cognitive ability after the President underwent a neurological screening. Jackson's performance during that extended news conference played a part in Trump's decision to nominate him to lead the VA, a White House official told CNN.

Tuesday, the Senate committee's top Democrat, Montana's Jon Tester, told CNN's Anderson Cooper that around 20 people had raised concerns to the committee about Jackson, and according to those people, Jackson would hand out prescriptions "like candy."

Tester described reports to the committee that alleged that on overseas trips, Jackson would "go down the aisle way of the airplane and say, 'All right, who wants to go to sleep?' And hand out the prescription drugs like they were candy ... and put them to sleep and then give them the drugs to wake them back up again."

"These are called controlled substances for a reason," Tester said on "Anderson Cooper 360."

The White House doctor also faced allegations of being drunk while overseas with then-President Barack Obama.

"If you are drunk and something happens with the President, it's very difficult to go in and treat the President," Tester said. "That's what multiple people told us, this was the case on several different trips."

Other concerns about Jackson included allegations of a toxic work environment, the senator confirmed.

"We were told time and time again the people above him he treated like gold, the people below him, he belittled, screamed at them, really created a very toxic environment to the point where the people who worked around him felt like they had to walk on eggshells because of his lack of respect for his job," Tester said.

In a letter to Trump sent Tuesday, the senators requested information "regarding allegations or incidents" involving Jackson dating to 2006, the year he was tapped to become White House physician.

One source on the committee told CNN prior to Jackson's withdrawal that lawmakers were also requesting information from the FBI, including Jackson's background check. Tester later said the FBI background investigation into Jackson was "clean."

Congressional sources also told CNN that committee aides had been in touch with individuals associated with additional allegations about prescriptions handled by the White House Medical Unit, which oversees the medical care of the president and administration aides.