President Donald Trump’s nominee to lead the Department of Veterans Affairs, Rear Adm. Ronny Jackson, is facing pressure to drop his bid amid concern that potentially damaging allegations from his past service could create an embarrassing public spectacle.

Two sources familiar with the proceedings on Capitol Hill tell The Daily Beast that lawmakers have been instructed to no longer push for Jackson’s confirmation after allegations surfaced that he was hostile to co-workers, struggled with alcohol use, and over-prescribed medications during his time in the Navy and during his tenure as the White House physician. A third source told The Daily Beast that lawmakers are particularly hung up on the fact that there are numerous allegations of repeat behavior on Jackson’s part—not isolated incidents.

Later Tuesday, Sen. Jon Tester (D-MT), the top Democrat on the Senate Committee on Veterans’ Affairs, confirmed the nature of the allegations. In an interview with NPR, he said more than 20 military employees contacted the committee about Jackson. Their allegations focused mainly on Jackson’s actions while on official business abroad.

“We were told stories where he was repeatedly drunk on while duty where his main job was to take care of the most powerful man in the world. That’s not acceptable,” Tester said, adding that Jackson was allegedly verbally “abusive” toward staff members and “belittle[d] the people underneath him.”

The senator later said in a CNN interview that Jackson was referred to as the “candy man” because he “handed out prescription drugs like they were candy.” Those substances included Ambien and Provigil, according to Tester.

Sen. Sherrod Brown (D-OH) told reporters “a significant number” of Jackson’s former colleagues had come forward to investigators and individual senators with concerns about Jackson in recent days.

“I don’t know what’s true, but I do know that all kinds of people are coming forward and that should have given the White House some pause of this nomination,” he said. “I don’t know if they are whistleblowers... They are people that knew him or know him, have worked with him in the military and former military that are willing to talk to investigators about him.”

He added, “Why wasn’t the White House doing that?”

Asked about his beleaguered nominee on Tuesday, Trump reaffirmed that he stood behind Jackson, calling him “one of the finest people that I have met.” But he all but openly pleaded with him to drop his bid, suggesting that the political pressure to do so was both untenable and unfair.

“If I were him—in many ways I’d love to be him—I wouldn’t do it,” Trump said when asked about the prospect of Jackson soldiering on.

Jackson’s Senate confirmation hearing, originally scheduled for Wednesday, was formally postponed on Tuesday as lawmakers continue to review the allegations—giving off the impression that the nomination was all but dead.

“I think it’s gone, unfortunately, because anytime things go backward like this and there is a postponement of a hearing, the support has come unwound,” said Rick Weidman, Executive Director for Policy and Government Affairs at Vietnam Veterans of America. “I’ve met him a couple times. I didn’t get a chance to say I know him well. People who knew him from his role at the White House say he is a great guy and a good physician. But that does not necessarily a good secretary make.”

Tester, the committee’s top Democrat, didn’t rule out that the hearing may be rescheduled, telling reporters it would be around ten days at the earliest since the Senate is not in session next week.

Tester said he still had information to review, including an FBI report which he planned to read Tuesday afternoon. He said later that the FBI file “appears to be fine.” Asked whether the nomination was still viable, Tester said: “We are still working on the vetting.”

While Tester hadn’t spoken with Jackson since the information came to light, Sen. Jerry Moran (R-KS), a member of the committee, met with the admiral in his office on Tuesday. There was no indication that Jackson was preparing to withdraw, Moran said. Jackson also addressed one of the allegations, according to Moran, who said Jackson assured him that he has never had a drink while on the job.

“To me, these are allegations—perhaps something that somebody said—but nobody has come forward to me or, as far as I know, nobody has sat in front of the committee and told us, ‘here are the problems with this nominee,’” Moran said.

At the White House, senior officials, including top aides on legislative strategy, were left scrambling on Tuesday. An emergency meeting was held in the morning to discuss next steps and how much pressure they should apply on the congressional side to salvage Jackson’s nomination, according to two administration officials. One official noted that while the position of the White House is to stand firm on Jackson for now, efforts to prop up him up could “implode” at any moment this week, given the internal concerns and their read of the political landscape.

Later Tuesday, the White House indicated it was sticking with Jackson, suggesting that the allegations against him were being driven by disgruntled former employees.

“He has improved unit morale, received glowing reviews and promotions under Republican and Democrat presidents, and has been given a clean vet from the FBI,” a senior administration official said. “He has never even been the subject of an Inspector General review and he will certainly not be railroaded by a bitter ex-colleague who was removed from his job.”

The administration pointed to former President Barack Obama’s handwritten praise of Jackson in his medical reports, in addition to a Controlled Substance Inventory Board audit of the White House medical team which it says “confirms Dr. Jackson’s medicinal prescriptions were completely appropriate.”

Jackson was an unexpected choice to lead the VA. He had never run a bureaucracy larger than a few dozen employees. What he lacked in experience, however, he made up with gravitas. Trump was reportedly smitten with how Jackson performed when discussing the president’s physical—which Jackson conducted—at the White House briefing room. And the pair got along well during their meeting to discuss the VA vacancy.

But Jackson had few allies outside of the president. The major veterans groups approached his nomination skeptically, and lawmakers on the Hill complained that they had not been consulted in advance and had limited understanding of Jackson’s managerial philosophies when it came to veterans’ care. Key players on veterans issues on and around Capitol Hill said they had never even met Jackson prior to his nomination, and found the manner in which he was chosen to be concerning.

“I have very serious questions that need to be addressed, and they should be addressed right now. Like today,” said Sen. Richard Blumenthal (D-CT), a senior member of the committee. “And the White House needs to be forthcoming. It has been really careless, maybe even negligent about vetting with a number of these nominations. And it probably needs to take steps to correct those deficiencies.”

Tester and Sen. Johnny Isakson (R-GA), the chairman of the veterans’ affairs committee, sent a letter to Trump on Tuesday asking for information on “improper conduct,” giving off few hints that they’re pining for a confirmation hearing to proceed.

“I’m very happy with what I know, and I think I’m pretty happy with what I don’t know,” Isakson told reporters.

Senate Majority Leader Mitch McConnell (R-KY), meanwhile, deferred to his deputy and the White House when asked about Jackson’s future.

“We’re going to wait and see what the administration and Chairman Isakson recommends,” he said.