Washington (CNN) Democratic presidential candidate Michael Bloomberg said Tuesday that he was "probably wrong to make the jokes," referencing allegations of sexist and misogynistic behavior made against him.

At last week's Democratic primary debate, Bloomberg had said women who made complaints against him in the past maybe didn't like a joke he had told . CBS's Norah O'Donnell followed that up at Tuesday's debate by asking, "Did these women take your jokes wrong, or were you wrong to make the jokes?"

"Probably wrong to make the jokes," Bloomberg responded.

The billionaire businessman and former New York City mayor said he doesn't remember what he said, but that "if it bothered them, I was wrong, and I apologize. I'm sorry for that."

After being repeatedly pressed last week by rival Sen. Elizabeth Warren of Massachusetts, Bloomberg said Friday that women covered by three nondisclosure agreements who had made complaints against him could be released from those pacts if they contacted his company. Bloomberg also said that moving forward, his company and presidential campaign would not offer confidentiality agreements to resolve claims of sexual harassment or misconduct.

After urging him to release women from the contracts at last Wednesday's debate, Warren also had brought a copy of a contract she had written herself to a CNN town hall the next night and urged Bloomberg to sign it so the women could tell their side of the story.

"The senator has got it, and I don't know what else she wants us to do. We're following exactly what she asked to do," Bloomberg said of Warren at Tuesday night's debate.

"The trouble is with this senator enough is never enough," Bloomberg said at Tuesday's debate.

"We did what she asked," he continued. "And, thank you, we've probably made the world better because of it. And by my company renouncing using these, we probably changed, hopefully, the corporate landscape all across America."

Bloomberg's comments came after Warren said at the debate that she had been fired when she was a visibly pregnant 21-year-old teacher.

"At least I didn't have a boss who said to me, 'Kill it,' the way that Mayor Bloomberg is alleged to have said to one of his pregnant employees," Warren said. Bloomberg replied that he had not made that comment.

Bloomberg faces claims from the 1990s that he made sexist and crude comments in the workplace. According to allegations laid out in two lawsuits, he told a group of female employees prior to a male colleague's wedding to "line up to give him a blow job as a wedding present," would regularly direct comments like "look at that nice piece of ass" at women in the office and upon learning that a female employee was expecting a baby, responded: "Kill it!"

Bloomberg previously has denied the making the "kill it" comment and other remarks laid out in at least two lawsuits, but has said he made comments that do not align with his values.