Andrew Puzder will stay on as CEO of his fast food company after withdrawing as President Donald Trump's choice for labor secretary.

A spokesman for Puzder told CNNMoney on Thursday that he would return to work as the top executive of CKE Restaurants, which owns Hardee's and Carl's Jr., after taking a "short break to re-charge with his family."

Puzder took himself out of contention for the secretary job a day before his scheduled confirmation hearing in the Senate.

Related: Andrew Puzder withdraws as a labor secretary nominee

Confirmation was becoming increasingly unlikely. A senior GOP source told CNN that four Republicans would vote against Puzder, and more were considering a no vote.

Puzder faced fierce opposition from Democrats from the start. He has fought against government regulation, a $15 minimum wage and the Affordable Care Act.

He also once employed an undocumented immigrant as a housekeeper. And in recent days, a domestic abuse accusation threw Puzder's nomination into turmoil.

Decades ago, Puzder's ex-wife, Lisa Fierstein, brought abuse charges against him, which she later dropped.

She appeared in disguise on Oprah Winfrey's TV show to talk about domestic violence in 1990. During the episode, the tape of which was made public by Politico, Fierstein said Puzder told her, "I will see you in the gutter. This will never be over. You will pay for this."

Senators privately reviewed the tape before Puzder's scheduled hearing.

Puzder has denied he abused his wife. In a letter to senators obtained by CNN, Fierstein said she regretted accusing Puzder of abuse during a rocky marriage.

--CNN's Dan Merica and Manu Raju contributed to this story.