The White House physician, Rear Adm. Ronny Jackson, who withdrew his nomination to lead Veterans Affairs, will not return as President Trump’s personal doctor, a report said Sunday.

He will be replaced by Sean Conley, a Navy officer who has filled in for Jackson over the past month, Politico reported.

Jackson withdrew his nomination last week after Sen. Jon Tester, ranking Democrat on the Veterans Affairs Committee, released documents that alleged the physician drove drunk, drank on the job, prescribed pills so freely he was known as the “Candy Man,” and created a hostile work environment.

Jackson has denied the claims, as have the White House and Secret Service.

At a rally Saturday in Washington, Mich., Trump blasted Tester for raising the allegations against Jackson and threatened to reveal some dirt on the Montana senator.

“I know things about Tester that I could say, too,” Trump said. “And if I said them, he’d never be elected again.”

Jackson, the physician for Presidents Barack Obama and George W. Bush, made headlines in January when he deemed Trump to be in “excellent” health after the annual examination.

Speaking of the president’s genetics, Jackson said, “Some people have just great genes. I told the president that if he had a healthier diet over the last 20 years, he might live to be 200 years old.”

Trump named Jackson to replace David Shulkin, the former VA secretary, in March after the department’s inspector general released a report faulting Shulkin for a trip he took to Europe last year with his wife, whose travel expenses were paid with taxpayer funds.