WASHINGTON — Ronny Jackson, President Donald Trump's nominee to lead the Department of Veterans Affairs, has grown frustrated with the process and is in active discussions with senior White House officials about withdrawing his name from consideration for the cabinet post, two sources with knowledge of the conversations said.

A decision about Jackson's future could come as early as Wednesday night or Thursday, one source said.

The development comes as congressional Democrats released additional details Wednesday on allegations of improper conduct involving Jackson who said earlier in the day that he has no intention of backing out.

Democrats on the Senate Committee on Veterans' Affairs spoke to nearly two dozen current and former colleagues of Jackson — most of whom the committee said are still in uniform — and said those interviewed "raised serious concerns about Jackson’s temperament and ethics, and cast doubt on his ability to lead the second-largest agency in government."

According to a two-page summary of those interviews released by the panel’s staff Wednesday, the White House medical unit had "questionable record keeping" for the medications it was distributing under Jackson's leadership. On one occasion, Jackson was said to have provided a "large supply" of the opioid drug Percocet to a staffer for the White House Military Office, which threw the office "into a panic" because it didn't know where the drugs went.

In addition to concerns about his prescribing practices, the summary noted "multiple incidents of drunkenness on duty," including one instance when Jackson could not be reached when he was needed "because he was passed out drunk in his hotel room." It added that Jackson once got drunk at a Secret Service going-away party and "wrecked a government vehicle." It did not note the year of the incident.

Jackson denied that specific allegation Wednesday, telling NBC News, "I never wrecked a car," and said that should be pretty easy to prove. He also claimed that he did not know where the allegations were coming from, and added that he is "still moving ahead as planned" with his nomination.

Let our news meet your inbox. The news and stories that matters, delivered weekday mornings. This site is protected by recaptcha

Earlier in the day, the White House rallied around Jackson as the embattled doctor continued to face allegations of impropriety.

Marc Short, Trump’s top legislative affairs aide, told reporters outside the White House that Jackson deserved a fair hearing before the Senate Committee on Veterans' Affairs and said the allegations against the doctor were politically motivated. He also indicated that Jackson has no plans to withdraw.

"We all look at allegations, but I think you have to question if there is a process within the Department of Defense for active military people to bring those concerns," Short said, noting that concerns were instead raised to the ranking member of the Veterans' Affairs Committee, Sen. Jon Tester, D-Mont., who has been vocal in sharing his findings.

Short added he thought it was unfair for Tester to have dubbed Jackson, the White House physician who shared the results of Trump's annual physical with the press earlier this year, the "candy man" — a reference to his prescription practices — in media interviews.

"Every year they come in and they do a review of the White House physician’s office on things like prescriptions, and every year they have said that he is totally in compliance with what he has been prescribing," Short said.

According to the summary of interviews released Wednesday, multiple people said Jackson was known as the "candy man" among White House staff because they said he would provide prescription drugs without paperwork.

Jackson also engaged in a pattern of handing out Ambien to help people go to sleep on Air Force One and distributing Provigil to help people wake up, says the summary, which added that he did not triage patients' medical history when giving those drugs out.

White House Press Secretary Sarah Sanders said Wednesday that Jackson's character had been praised by three administrations and that he underwent the standard vetting process before Trump selected him to lead the Department of Veterans Affairs. She said she is "not aware" of any allegations made against the doctor prior to his nomination.

Meanwhile, Trump — who showed support for Jackson a day earlier, despite opening the door for him to withdraw his nomination — met with Jackson for a private Oval Office meeting Tuesday, and asked his nominee how he wanted to address the allegations, according to a White House official.

The official told NBC News that Jackson told Trump that the allegations are untrue and misleading and that he wants to make his case, which the president encouraged him to do.

Tester defended releasing the summary of allegations in an interview Wednesday, saying that the document was also shared with the office of the committee's Republican chairman, Sen. Johnny Isakson.

"So we gave it to them and then we gave it to other people and you guys need to have it too," Tester said on MSNBC's "Meet the Press Daily," adding that his staff was sharing information as quickly as possible with his GOP counterpart.

"These aren't my accusations, these are accusations from retired and active-duty military personnel that have come to us. We are just trying to follow up and see what is true and what is not," Tester said.

Jackson's confirmation hearing had been scheduled for Wednesday, but was postponed amid concerns over the allegations.

Asked Wednesday whether the White House is pushing to reschedule the hearing, Sanders told reporters, "We are continuing to work with members on the Hill." Tester also did not rule out the possibility that Jackson could still receive a hearing.

Sen. Jerry Moran, R-Kan., said Jackson denied allegations of a hostile workplace while working as the White House physician, and told Moran that he has never had a drink while on duty.