Ronny Jackson has vehemently denied all the allegations against him ever since President Donald Trump nominated him to lead the Department of Veterans Affairs. But after withdrawing his name from consideration for the job, he won’t be returning to his previous role as the president’s personal physician, Politico reports, citing two unnamed “senior administration officials.” Sean Conley, who took over the role last month, will reportedly stay in the job.

Jackson, who also served as physician for George W. Bush and Barack Obama, withdrew his nomination for secretary of veterans affairs last week following numerous allegations against him. Among the claims against Jackson were accusations that he was freewheeling in prescribing pain medication, got drunk on the job, and created a hostile work environment. After he withdrew his name from consideration, Jackson returned to the White House Medical Unit, but not as the president’s physician.

Trump has defended Jackson and called on Montana Sen. Jon Tester, the top Democrat on the Veterans’ Affairs Committee, to resign for his role in publicizing the allegations against Jackson. During his Saturday rally in Michigan, Trump lashed out against Tester and suggested he had compromising information on the senator. “Tester started throwing out things that he’s heard. Well I know things about Tester that I could say too,” Trump said. “And if I said them, he’d never be elected again.”

"They try to destroy a man."



Moments ago, President @realDonaldTrump discussed @jontester and Dr. Ronny Jackson. pic.twitter.com/VdSuOjNsPB — Fox News (@FoxNews) April 28, 2018

Although Trump has focused his ire on Tester, several Republicans have also raised questions about Jackson’s past. On Sunday, House Oversight Chairman Trey Gowdy told CBS’ Face the Nation that “some of those allegations do warrant being investigated.”