Washington (CNN) In the end, Andrew Puzder had too much baggage -- both personal and professional -- to be confirmed as President Donald Trump's Cabinet.

Puzder withdrew his name from contention to be Trump's pick to lead the Labor Department on Wednesday, a move that ended weeks of negative stories about the fast food executive. The relentless drumbeat of negative press -- including stories about his divorce, how he employed an undocumented immigrant for years and scrutiny into his business practices -- wore on Puzder, people close to him said.

Republicans understood that any labor secretary nominated by Trump would have been attacked by Democrats and their union allies. But even they were surprised at the breadth of attacks against Pudzer.

It was the fact that Puzder had no easy path towards the nomination, though, that was the last straw for the nominee. Senate Republicans told the White House on Wednesday that Puzder was losing support, urging the Trump administration to pull his nomination. Once word got to Puzder and his team, the decision was made to begin to cancel his confirmation commitments and, eventually, drop out all together.

"After careful consideration and discussions with my family, I am withdrawing my nomination for Secretary of Labor," Puzder said. "I am honored to have been considered by President Donald Trump to lead the Department of Labor and put America's workers and businesses back on a path to sustainable prosperity."

Puzder's nomination, which the Trump transition team announced in early December, was immediately met with criticism from labor unions and Democrats.

While Trump said in a statement at the time that he would "save small businesses from the crushing burdens of unnecessary regulations that are stunting job growth and suppressing wages," Democrats and associated groups began to dig in, believing early on the Pudzer was one of the most vulnerable Trump cabinet nominees.

Christine Owens , executive director at the National Employment Law Project, laid out the battle lines on Puzder's business practices this way: "It's hard to think of anyone less suited for the job of lifting up forgotten workers than Puzder, a billionaire CEO who vocally opposes any meaningful increase in the minimum wage, who talks glibly about replacing workers with machines, and who consistently attacks rules that protect both workers and law-abiding employers."

Shortly after his nomination, though, it began to grow clear -- both to people inside his camp and Democrats outside -- that a key battle in Puzder's fight would focus on his treatment of women, both those he employed as an executive and the woman he divorced in the 1980s.

The RiverFront Times, a paper in St. Louis, reported on December 8 -- shortly after he was nominated -- that Puzder was accused of domestic abuse during 1986 divorce proceedings. The story grew from there, leading mainstream media outlets to check with the lawyer, who told Time magazine that he found Lisa Henning, Puzder's ex-wife, to be "to be credible and believable."

The personal nature of the attacks against him began to take a toll on Puzder, people close to the former Labor nominee told CNN. Puzder, these sources said, viewed his divorce as a private, personal matter that shouldn't be exploited by his opponents.

Henning would later walk back her charges against her ex-husband, but the damage was already done. The divorce storyline continued, propelled by the fact that Henning had appeared on "The Oprah Winfrey Show" in disguise at the time to recount her abuse story. The video of the interview was eventually shown to senators and went public Wednesday, the day Puzder withdrew his nomination.

At the same time, Democrats privately began drawing up plans to hold Puzder's feet to the fire on comments he made about women and racy ads run by Carl's Jr, the company he oversaw for years. Puzder defended the ads, which featured scantily-clad models eating burgers while washing cars, but the storyline, pushed by Democrats, was meant to raise questions about the nominee's character.

"It certainly raises questions for Democrats that the nominee for Department of Labor would not only advocate for harmful stereotypes about women, but go so far as to say they are at the core of his company's values," a Democratic Senate aide told CNN in December. "And given the role that the secretary of labor plays in standing up for women's rights at work, Republicans should be concerned as well."

As the drumbeat of negative stories continued, Puzder grew wary and was taken aback by the harshness of politics, a business ally and GOP sources told CNN in January.

"He may be bailing," said a Republican source plugged into the Trump transition effort. "He is not into the pounding he is taking, and the paperwork."

Puzder and his team maintained that he was looking forward to his hearing, but behind the scenes, there was more apprehension on the path forward.

Puzder's initial hearing with the Senate Committee on Health, Education, Labor and Pensions was delayed, in part because his ethics paperwork and financial disclosure form had yet to be approved by the Office of Government Ethics.

Puzder suffered another significant hit in early February when he was forced to admit that he had employed an undocumented immigrant for years, an admission that in the past has sunk other Cabinet secretary nominees.

"My wife and I employed a housekeeper for a few years, during which I was unaware that she was not legally permitted to work in the US," Puzder said in a statement. "When I learned of her status, we immediately ended her employment and offered her assistance in getting legal status."

Though Puzder said he and his wife paid back taxes on the employee, the damage was done and Democrats set in, calling on the nominee to withdraw by noting how past Democrats had withdrawn over the same offense.

"They ought to withdraw Mr. Puzder before he further embarrasses this administration and further exposes the hypocrisy of President Trump, who says one thing to the American worker and does another," Senate Minority Leader Chuck Schumer said on February 9.

In the face of all this, however, Puzder's nomination lurched along. The nominee's spokesman, George Thompson, called Schumer's comments "fake news."

Eventually, weeks after Puzder initial hearing was set to take place, the ethics office approved the businessman's paperwork, requiring him to liquidate all of his holdings in two companies before he assumed his role at the Department of Labor.

The Senate committee tasked with considering Puzder then set a hearing date: February 16.

The movement, though, seemingly kicked opposition to Puzder in overdrive, with liberal groups marshaling significant support to sink the nomination and keep all Democrats in line against the fast food executive. These groups, including labor unions, organized pseudo call centers to hammer senators thinking about approving the nominee.

Republicans in the Senate responded with an aggressive effort to save Puzder's embattled nomination, leaning on well-funded business groups, the White House and Mitch McConnell, the powerful Senate majority leader.

McConnell was Puzder's top cheerleader on Capitol Hill, sources told CNN, telling fellow senators that the nominee would be more prepared than any nominee in history to be labor secretary.

McConnell's efforts, though, didn't end up being enough.

The balance fully tipped against Puzder when four Republican senators -- Susan Collins of Maine, Lisa Murkowski of Alaska, Tim Scott of South Carolina and Johnny Isakson of Georgia -- told their leadership that they were withholding their support for Puzder until they get more information. The comments sent shockwaves around Washington, including in the White House, where aides tasked with shepherding Trump's nominees through the Senate spent long nights trying to lock up Republican support.

The effort failed, and Puzder eventually began to recoil from continuing to fight.

At first, Puzder told the White House the he wanted to withdraw if they felt there was no path forward. His team then began to cancel their plans, including a grueling prep session Puzder was set to undergo on Wednesday afternoon.

After Puzder officially withdrew, the finger-pointing began in earnest, with much of the blame falling on industry groups that Republicans feel did not back up the fast food executive.

Republican sources said they were swamped by labor groups and progressive forces looking to derail Puzder, while the business community sat on the sidelines.

"There was no campaign to support him or defend him," a source close to Puzder told CNN. "It is unfortunate that the industry did not step up to run a modern day campaign."

Another Republican source put it more bluntly when blaming business groups: "His entire support network made one TV ad that looked like a welcome video for new Hardy's employees. ... It's no wonder the unions eat their lunch every time out."