(CNN) Dr. Ronny Jackson exuded confidence when he emerged from a Capitol Hill office Tuesday evening, telling reporters that he looked forward to answering tough questions from lawmakers about his past.

By Thursday morning, Jackson was back in the White House basement, returning to his job as the President's physician after his nomination to become the next Veterans Affairs secretary was scuttled.

In less than 48 hours, Jackson was downgraded from a well-respected physician who served presidents from both parties to the latest casualty of Washington's scandal machine. Not since former President George W. Bush's 2005 attempt to elevate Harriet Miers to the Supreme Court has a high-level nominee been so thoroughly humiliated in the confirmation process.

But the process was especially ugly -- and swift -- for Jackson. His downfall was fueled by a laundry list of serious, but uncorroborated, allegations from nearly two dozen current and former colleagues that ultimately came together in a memo released by Senate Democrats with, according to sources, the tacit approval of Republicans.

This account of the final hours before Jackson withdrew his nomination Thursday morning is based on more than a dozen interviews with lawmakers, administration officials, congressional sources and veterans' groups. It depicts Jackson's surprise path to the nomination, the bipartisan concerns about the White House physician's conduct and the Trump administration's mad scramble to salvage a nomination that was, in many ways, plagued by self-inflicted wounds.

The interviews show that 15 months after taking office, the Trump administration is still struggling to professionalize its nomination process to ensure vetting is taken seriously and lawmakers on Capitol Hill are prepared for any surprises.

"We did him no favors," a White House official said Thursday, acknowledging the administration did not fully appreciate the magnitude of the task and the campaign that would rise up against Jackson's nomination.

The White House didn't respond to a request for comment on this story.

Surprise selection

Jackson's selection as the nominee to succeed embattled former VA Secretary David Shulkin came as a stark surprise to him late last month when the President summoned him to the Oval Office and asked him to take on the task of leading the sprawling agency. Jackson knew about the decision only a day or two before it was made public, an official said, a sign that inadequate vetting had taken place.

On Capitol Hill, lawmakers in both parties openly questioned the President's decision to tap Jackson, one largely made out of personal affinity, and were skeptical that he had the experience to lead an agency that is known as a management boondoggle.

It wasn't Jackson's background that initially worried White House chief of staff John Kelly and others in the West Wing, but rather his lack of experience for the job. Kelly gently argued against the Jackson pick, an official said, but the President's mind was made up — and there was no obvious and immediate alternative.

After the White House officially sent his nomination to the Senate, Jackson appeared to be doing all the things a Cabinet nominee would typically do. He met with lawmakers privately to answer questions and submitted routine paperwork to the committee of jurisdiction, in this case the Senate Committee on Veterans Affairs.

But White House officials said that Jackson had far less help preparing for his confirmation than Cabinet nominees typically get, with one source telling CNN that the White House was "stretched thin" due to the other nominations making their way through the Senate, including Mike Pompeo's nomination for secretary of state and, his replacement at the CIA, Gina Haspel.

The source said three lower-level aides from the White House communications shop and a few others from the Legislative Affairs Office helped oversee preparations for Jackson's nomination.

"But that was it," the source noted, adding that the communications staff at the VA "basically sat on their hands" and refused to help throughout the process because they knew they could lose their jobs once a new secretary took over the agency.

Bipartisan concerns grew on Capitol Hill earlier this week about Jackson's conduct on the job. And the White House moved to defend Jackson against what they called false accusations. Jackson indicated repeatedly to reporters that he intended to stay the course even as negative headlines mounted.

Speaking at a Tuesday news conference alongside the president of France, President Donald Trump robustly defended Jackson as "one of the finest people I have met," but said that Democrats had launched an unfair attack on the White House physician's record.

He also hinted that Jackson may well not continue in the process -- and seemed to give him a window to leave.

"I don't want to put a man through a process like this," Trump said. "The fact is, I wouldn't do it. What does he need it for?"

On Tuesday, the top senators on the Veterans Affairs committee announced that they would postpone a confirmation hearing for Jackson scheduled the next day, pending further information. And they took an unprecedented step of writing a letter to the President, requesting "any and all communication" between the Pentagon, the White House Military Office and the White House Medical Unit "regarding allegations or incidents" involving Jackson dating back to 2006, the year he was tapped to serve as a White House physician.

The next day, Senate Democrats released an explosive two-page memo, based on the account of nearly two-dozen of Jackson's former and current colleagues, about his conduct, adding more detail to the accusations that led the committee to delay his confirmation hearing and further casting doubt that his nomination could survive.

Jackson's increasingly dire situation

Jackson's situation grew more dire over the last 24 hours.

Republicans on the Veterans Affairs committee were planning to begin increasing pressure on the White House to withdraw the nomination, according to a source briefed on the matter. They concluded that there was no path for Jackson to be confirmed. In private discussions with senators, Jackson himself was told that his confirmation prospects appeared to be waning given the narrow GOP majority in the Senate and the likelihood several Republicans would oppose him, the source said.

Making matters worse for Jackson, Republicans were unwilling to offer a public defense of Jackson. And they didn't know whether to believe his denials.

"He didn't say anything that didn't seem believable," said Sen. Jerry Moran, a Kansas Republican who sits on the committee, referring to his meeting this week with Jackson, who denied doing anything wrong. "As it's so evident to me and others across the country, the question is: 'Who do you believe?'"

Georgia Sen. Johnny Isakson, the GOP chairman of the panel, had been kept apprised of ranking member Jon Tester's investigation over the last several days -- and viewed the sources of the allegations as serious enough to warrant a deeper probe.

Isakson told CNN he "never" told Tester to back off, or to avoid disclosing the allegations to the news media. "Everybody has to make their own decisions on that," Isakson said.

Last week, Tester's office began hearing from current and former employees of Jackson, raising alarms about his professional conduct. Suddenly, the calls began pouring in, with staff interviewing sources until 9 p.m. at one point. After interviewing about 25 people, Tester's office was still trying to track down key details, including the car that was apparently crashed when Jackson allegedly was driving while intoxicated, sources said. Jackson denied the allegation to reporters on Wednesday.

Tester continued to keep Isakson apprised of his investigation and his plans to go public. Isakson did not stop him.

"This was totally organic," Tester told CNN of the allegations that came his way. "We didn't have an advertisement. We weren't walking around saying anything about Admiral Jackson. We were concerned about, like I said, his ability to run an agency as big as the VA. This came at a different angle."

Privately, Republicans on the committee were perplexed. They couldn't find documentation from his past background checks that corroborated the allegations. But they came from credible sources that couldn't be ignored.

"I don't have a problem with the way it was completed," said South Dakota Sen. Mike Rounds, a Republican member of the committee, referring to Tester's probe. Rounds had planned to meet with a senior White House official Thursday morning to ask questions about Jackson. "These are serious allegations -- unsubstantiated but serious."

Moran, the Kansas Republican, said he met with Jackson after the initial round of allegations were reported in the media. Jackson indicated he had done nothing wrong.

"I asked him if there were allegations out there, what would you want to tell me about that?" Moran recalled. "We talked about alcohol. The answer was, 'I've never taken a drink on duty.' We talk about prescription drugs, and he said it is highly regulated by the pharmacy."

Moran said that Jackson told him that he "might have a beer with the security crew when I'm not on duty."

Moran still was unsure.

"I indicated that I need to be convinced that this is the right person who could do a solid job," Moran said, referring to how he felt the moment that Jackson was nominated. "I had not reached that conclusion, and the more recent stories arrived and it became more complicated."

In the end, the respect and admiration for Jackson clouded the judgment of those in the West Wing. Trump wanted to offer Jackson a final lifeline on Wednesday by praising him to reporters — at least to help clear his name and rally some Republicans around his beleaguered nominee — but several advisers argued against it.

As late as Wednesday evening, Jackson was still working Capitol Hill in search of support for his nomination. But before sundown, Trump increasingly believed Jackson should step aside "before things get worse," an official familiar with the talks said.

Jackson returned to the White House Wednesday evening and told staffers he wanted to pull out, according to a White House official, saying that the scrutiny was too much for him and his family to bear. Jackson, the official said, was taken back by the vitriol of the process, even as he maintained that the mounting allegations against him were not true.

Jackson emerged from senior staffer's offices late Wednesday evening and told a group of waiting reporters: "Look forward to talking to you guys in the next few days."

By the time Jackson left the West Wing late Wednesday night and headed home, the plan was set to send out the announcement of his withdrawal around breakfast time Thursday, sparing him another day of twisting in the wind.

Inside VA headquarters

As the President's nominee to lead the agency seemed increasingly on the rocks, officials at VA's Vermont Avenue headquarters were working to show that even without a permanent secretary, things were running smoothly. The agency's current interim leader is Robert Wilkie, who also serves as a Defense Department undersecretary.

In a statement Wednesday, VA spokesman Curt Cashour said that since Shulkin was ousted from the department in March, "senior VA officials are now on the same page, speaking with one voice to veterans, employees and outside stakeholders," and that employees who were "wedded to the status quo and not on board with this administration's policies or pace of change have now departed VA."

But with Jackson's nomination scuttled, the White House is now back at the starting line to find a new nominee, leaving the Department of Veterans Affairs in a state of turmoil. In just the last four years, the agency has had two confirmed secretaries and three interim secretaries.

The President hinted on Thursday morning that he is already considering who could replace Jackson, and a senior White House official confirmed that was the case. Trump said during an interview on "Fox and Friends" that he might seek someone with more "political capability" than Jackson.

Another White House aide suggested that two names already in the mix are Ike Perlmutter, the Marvel Entertainment CEO who is a member of Trump's advisory board on veterans affairs, and Toby Cosgrove, a Vietnam War veteran, heart surgeon and former CEO of the Cleveland Clinic.

A former White House official who worked on veterans' issues suggested that Trump may consider Wilkie, a veteran who has previously been confirmed by the Senate, to lead the department on a permanent basis. But the leaders of some veterans' groups say he should have never been brought over to lead VA to begin with.

"We now face the prospect of a stunning eighth nominee for VA secretary since 9/11," said Paul Rieckhoff, the founder of Iraq and Afghanistan Veterans of America. "It's been an unprecedented time of chaos, political agendas and uncertainty. And millions of veterans and their families have paid the price."

Perlmutter and Cosgrove declined to comment for this story.

Reporting for work

Jackson reported for work early Thursday morning, before the decision that he would not continue the nomination process was public, wearing a red tie and a gray suit, not the military uniform he occasionally wears while working and that he donned as he strode between meetings with senators on Capitol Hill in the days prior.

A person who spoke to Jackson on Thursday morning described him as frustrated at allegations he claims were taken out of context, or were altogether false. While Jackson told confidantes he would have been willing to fight for his nomination, it was becoming increasingly clear that doing so would damage his reputation further.

As the White House announced that Jackson was withdrawing his name from consideration, Jackson was fetching a coffee from the White House mess.

Asked about his future, White House press secretary Sarah Sanders said that "Admiral Jackson is a doctor in the United States Navy assigned to the White House and is here at work today."

Meanwhile, preparations were underway a floor above for an event honoring wounded veterans -- a ceremony Jackson was originally slated to attend, but skipped after his nomination was pulled, according to a person familiar with the planning.

Without a Veterans Affairs nominee to tout, Trump instead lauded Wilkie, the interim leader. And he boasted he'd cleaned out the agency's bad actors.

"We have people that were terrible working there and they're gone," he said.