Drinking too much on the job. Overseeing a “toxic” work environment. Overprescribing medication. Wrecking a government vehicle. Knocking on a female colleague’s door loudly enough to attract Secret Service attention.

Those are serious allegations against any physician — but it’s worse when they’re leveled against Dr. Ronny Jackson, the top White House doctor and President Donald Trump’s pick to lead the Department of Veterans Affairs. And now, new details are emerging that could further imperil his chances of making it through what could be a bruising Senate confirmation fight.

Jackson’s woes began on Monday night when lawmakers said there were issues with his job performance (besides serving as Trump’s personal physician, Jackson is also an active-duty Navy admiral). Hours later, reporters learned of the drinking, work environment, and medication dispensation allegations.

Things got tougher for Jackson on Wednesday when the New York Times published details from a memo alleging that the doctor had a pattern of substance abuse and careless prescribing of drugs. Democratic staffers working for the Senate Veterans’ Affairs Committee compiled the document as part of their background investigation of Jackson ahead of potential confirmation hearings.

According to the memo, which was later published in full by the Washington Post, Jackson wrote himself prescriptions before getting caught, provided a “large supply” of the opioid Percocet to a White House military aide without filling out paperwork, and destroyed a government vehicle while drunk.

The memo also alludes to an incident from a 2015 overseas trip where Jackson allegedly banged so loudly on a female colleague’s door while drunk that the Secret Service stepped in so he wouldn’t wake the president.

If any of the allegations are true, he could lose not only the chance to lead the VA but also much of his sterling reputation.

It was already not looking good for Jackson, whose surprise nomination in late March had prompted immediate concerns about whether he had the proper level of experience to run the sprawling VA. Jackson currently manages a staff of 70; the VA employs 375,000 people and has a budget of more than $185 billion.

On Tuesday morning, the two leaders of the Senate Veterans Affairs Committee — Sens. Johnny Isakson (R-GA) and Jon Tester (D-MT) — indefinitely postponed Jackson’s hearing, which was previously scheduled for Wednesday. They also sent a letter to the White House asking for any documents “regarding allegations or incidents involving Rear Admiral Jackson from 2006 to present.” During that time, Jackson led the White House Medical Unit and served as the physician to three presidents — George W. Bush, Barack Obama, and Trump.

It’s unclear if Jackson, who met with Isakson on April 16, disclosed any information about his alleged misconduct to the committee chair. But Isakson has reportedly called the White House in recent days to let them know about growing concerns on the Hill over the allegations.

Jackson is already on the defensive. In a Tuesday meeting with Sen. Jerry Moran (R-KS) — a Veterans Affairs committee member — Jackson stated he never drank while on duty, the senator said.

And during a press conference alongside French President Emmanuel Macron on Tuesday, Trump said he was unaware of any allegations against Jackson, but that “it’s totally his decision” whether to remove himself from consideration. “I don’t want to put a man through a process like this,” he added. “It’s too ugly, and it’s too disgusting.” Those comments potentially provide cover for Jackson to withdraw himself and save the White House from having to pull the nomination or see it defeated.

If that happens, it would be a startling change of fate for a nominee who days ago seemed bound to become the next veterans affairs secretary. And while Jackson is the only one to blame for his alleged misconduct, this is also very much a crisis of Trump’s own making.

How Jackson got the VA secretary nomination

Trump isn’t the only president to have had warm feelings for Jackson, who often went the extra mile for his patients. An April 19 Washington Post profile of Jackson noted:

He helped George W. Bush clear brush at his Texas ranch. He supplied Barack Obama with Nicorette gum even as he urged him to quit the nicotine substitute. He once was so eager to deliver a sling to Vice President Richard B. Cheney for a sore arm that his sprint toward the presidential helicopter caught the attention of Secret Service agents, a friend said.

Jackson seemingly got the nomination because he went the extra mile for Trump too. He spent more than an hour last January answering questions about the president’s mental and physical health. During that press conference, the Navy doctor famously said “if [Trump] had a healthier diet over the last 20 years, he might live to be 200 years old.”

Jackson already had a good rapport with Trump — as the top White House doctor, he has to travel and spend time with the president — but his performance at this press conference made him a star in Trump’s eyes.

On March 28, Trump tweeted his intent to nominate Jackson to lead the VA after reportedly pushing out former Secretary David Shulkin over disagreements about how best to provide medical care for veterans. (Basically, Trump wants to privatize that care; Shulkin wanted to reform the way the government provides services. They battled over that issue for months until Trump finally removed him.)

But it turns out that Jackson received almost no vetting before Trump’s tweet. Had he gone through the normal vetting process, it’s likely the administration would have gotten at least some warning signs about his possible misconduct before making the pick.

Jackson was seen as a strange choice from the start. Before joining the White House, he was a top combat surgeon, at one point overseeing dozens of staffers in Iraq. Now he leads a small office of around 70 employees with a minimal budget and a simple mission: treat the president, the first family, and others at the White House.

Leading the Department of Veterans Affairs, though, is an entirely different job. It employs around 375,000 people with a budget surpassing $185 billion. Only the Defense Department is a bigger US government agency. Senate Republicans quietly grumbled that Jackson didn’t have the management experience required to run such a large bureaucracy.

Trump acknowledged Tuesday that “I know there’s an experience problem” with Jackson. Whether that means Trump will pull the nomination is a very different story since he appears to value two things above all: loyalty and sycophancy. Jackson’s chumminess with the president, and the praise the doctor lavished on him in January, could persuade Trump to try to push him through what would almost certainly be bruising Senate confirmation hearings.

Back in March, I asked Anthony Principi, the veterans affairs secretary from 2001 to 2005, how Jackson could best lead the VA. “He has to command the respect and restore morale and leadership to the department. That’s the most important challenge ahead of him,” he told me.

Should Jackson get through the confirmation process, he’ll start with a disadvantage on that front. But at this point, even getting a confirmation hearing will be hard enough.

Correction: Sen. Johnny Isakson called the White House to express concerns mounting on the Hill over the allegations against Jackson. A previous version said he suggested the White House might need to consider withdrawing Jackson’s nomination on the call.