(CNN) Senate Veterans Affairs Committee members are assessing allegations from whistleblowers that have told the panel about nominee Ronny Jackson's questionable behavior including excessive drinking and a "toxic" work environment under his leadership, according to two former White House medical staff members who have spoken with the committee.

Both sources who spoke with CNN told the committee about behavior they observed while working in the White House medical unit.

Lawmakers who spoke with CNN expressed worries that this could represent a pattern of behavior and not a collection of isolated incidents.

In addition, committee staff have been in contact with individuals associated with additional allegations regarding the handling of prescriptions by the White House Medical Unit and a workplace survey that was done because of issues in the unit under Jackson, according to congressional sources. The accounts come from former and at least one current associate of Jackson, according to the sources.

The accusations have roiled the nomination process leading to the postponement of a hearing scheduled for Wednesday. Wednesday's hearing. However the committee has not substantiated the claims, with little documentation available to back the claims.

Jackson told reporters on Tuesday that he "can answer the questions absolutely."

"I'm looking forward to rescheduling the hearing and answering everyone's questions," Jackson said Tuesday afternoon on Capitol Hill as he left a meeting with committee member Jerry Moran, a Kansas Republican.

Trump defended Jackson at a news conference Tuesday as "one of the finest people I have ever met," and said it is "totally his decision" on whether Jackson wants to withdraw.

Details from 2012

A 2012 Navy Medical Inspector General review of White House medical operations recommended the Navy "consider" replacing either Jackson or another military physician of equal rank, or both according to an administration official who is directly knowledgeable about the report.

The six-page report was ordered as a review of White House medical operations when staffers reported low morale because of bickering between the two men who were both Navy captains at the time. The official emphasized the report only recommended the change in personnel be "considered," but didn't make it a mandatory order.

It likely would have been up to the Obama White House to make a change if it wanted to, the official said. The two men did not apparently get along because they were of equal rank and struggled for control of the operation. The official said it became clear that "line of authority" needed to be established.

The report ended with a strong critique of the other doctor saying the medical staff believed he was "irrevocably damaged his ability to effectively lead and serve."

The report did not ultimately effect his promotion.

A senior administration official downplayed the accusation of a toxic work environment surrounding Jackson, saying a workplace survey was done in 2012 after the doctor was promoted and some people in his office complained about the work environment. That report didn't address alcohol or patient care, according to the official, who saw the document but declined to release it publicly.

The White House Medical Unit oversees the medical care for the President, his family and administration aides. It includes a team of doctors and nurses and operates from a suite of offices in the basement of the White House residence.

The Veterans Affairs Committee's top Democrat, Sen. Jon Tester of Montana, told CNN that the FBI background investigation into Jackson is "clean" -- and that there are no issues of concern in his background check. They are still inquiring about the complaints that have been made to the committee to see if they can be corroborated. The White House is only allowing the committee chair and ranking member to review the FBI background check, per a source familiar with the matter.

Hearing postponed

Lawmakers leading the confirmation announced on Tuesday morning that the hearings will be delayed indefinitely following allegations related to improper conduct in various stages of Jackson's career.

The Senate Veterans Affairs Committee Chairman Johnny Isakson and Tester of Montana, made the announcement on Capitol Hill. The two committee leaders said they want more information about the allegations of misconduct involving Jackson. They declined to discuss the nature of the allegations -- and both men stopped short of calling on him to withdraw.

Sen. Mike Rounds, a Republican from South Dakota and member of the VA committee, told reporters that over the weekend, Isakson held a call with GOP members on the VA committee to brief them on what he was learning about Jackson. Rounds said that it was clear from the call that Isakson and Tester had been communicating with each other about what they were learning.

CNN reported Monday that lawmakers from both parties on the committee are raising concerns about allegations involving Jackson and are reviewing them to determine if they are substantial enough to upend his nomination. Committee members have been told about allegations related to improper conduct in various stages of his career, two sources said.

Combine that series of allegations that one GOP senator told CNN would be "devastating" if true, with overall ambivalence that bordered on opposition by Republicans and things are not in a good place for Jackson.

Two sources with knowledge told CNN that Isakson called the White House at one point in the past few days to outline the depth of the concern raised about Jackson in the allegations. Those sources told CNN that while there was no explicit demand to pull the nomination, it was their understanding that message was intended. Isakson's office denied the senator ever directly suggested or intended to convey to the White House that they consider pulling the nomination.

Isakson called White House Chief of Staff John Kelly on Monday night to inform him Jackson's hearing would be postponed.

GOP aides that spoke with CNN Monday night were characterizing this nomination going down as a "when," not "if," situation.

Democrats huddled privately over these allegations Monday night.

The bigger issue: the allegations are a trigger, not the sole reason, for Republicans to bail on this nomination. For weeks committee Republicans have been grumbling about the White House dropping Jackson, who they viewed as unsuited for the job, on their respective laps.

Several GOP aides described their bosses as willing to hold off from criticism until the hearing in deference to the President's selection. Now they're confronted with these allegations and Jackson doesn't have many allies on the Hill willing to protect him.

Monday's news left lawmakers and their staff scrambling, as both sides are trying to figure out what's true. As one senior Republican aide put it: "A mess. A complete, utter mess."

But at least one Republican did not yet see this as the end of the line for Jackson's nomination.

Sen. James Inhofe of Oklahoma told reporters, "I think he's damaged right now but he's resurrectable."

This story has been updated to include additional developments and information.