WASHINGTON — To senior White House aides serving the last three presidents, Dr. Ronny L. Jackson was the war-tested doctor who served in Iraq, helped them cope with their high-pressure jobs and ran his medical staff with the rigor befitting his rank of rear admiral in the Navy.

But inside the White House medical unit, a military-run office with a few dozen doctors and nurses, Dr. Jackson was viewed as a bully and someone who kept sloppy medical records, drank too much and loosely dispensed strong drugs to curry favor with the powerful politicians and political aides he admired. Three current and former colleagues said that Dr. Jackson was sometimes intoxicated during overseas trips and that staff members were often ordered to leave a bottle of rum and a Diet Coke in his hotel room.

On Thursday, Dr. Jackson withdrew from consideration to join President Trump’s cabinet as the secretary of Veterans Affairs after those and other allegations about his behavior cascaded into public view. Dr. Jackson, Mr. Trump and numerous White House officials dating to the Obama presidency insisted none of it was true.

“These false allegations have become a distraction for this president,” Dr. Jackson said.

Dr. Robert G. Darling, a former White House physician to President Bill Clinton and a retired Navy emergency medicine doctor, said, “This is a guy who served this country, who is in this job pretty reluctantly, and now he’s getting hung out to dry.”