Herman Cain doesn’t intend to withdraw from consideration for the Federal Reserve Board, despite an increasing number of Republican senators vocalizing opposition to his nomination, the Wall Street Journal reports.

Four Senate Republicans have now joined Democrats in rejecting his potential appointment, effectively tanking it. But Cain doesn’t seem particularly interested in changing his plans.

“I don’t quit because of negative criticism,” he told the Wall Street Journal’s Paul Kiernan. “I don’t quit because of negative attacks. And I don’t quit because several senators have expressed reservations about my qualifications.”

Cain, a 2012 Republican presidential candidate and the former chief executive of Godfather’s Pizza, is one of two controversial candidates President Donald Trump recently floated as potential picks for the Federal Reserve Board. Neither Cain nor Stephen Moore, the other Fed candidate, has been officially nominated by the White House yet, but both Democrats and Republicans have already expressed concerns about the two.

Amid this pushback, Trump economic adviser Larry Kudlow has said he’s interviewing other potential candidates who could fill the openings Cain and Moore are being considered for, NBC News reports. “We’re talking to a number of candidates. We always do,” Kudlow told reporters.

Cain faces sexual misconduct allegations that some lawmakers see as a roadblock

Democrats argue that Cain, who has founded a super PAC dedicated to backing Trump and faced four allegations of sexual misconduct, isn’t the right person to serve on the board of the world’s most powerful central bank. And they’re far from alone.

At this point, Sens. Cory Gardner (R-CO), Lisa Murkowksi (R-AK), Mitt Romney (R-UT) and Kevin Cramer (R-ND) have all said they won’t be voting in favor of Cain’s confirmation, with Cramer previously telling the Hill that sexual misconduct allegations Cain has faced were the deciding factor for him.

“His showmanship doesn’t bother me, his business experience I think is great, simplifying the tax code is fine by me, but character still does matter,” he said.

Cain would need at least 50 votes to be confirmed in the Senate (with Vice President Mike Pence as a tie-breaker), and now that four Republicans, of the party’s 53-person majority, have stated their opposition, he’s expected to fall short of that threshold.

Beyond these four lawmakers, there also appeared to be broader concerns among the GOP regarding a Cain appointment. Majority Leader Mitch McConnell had even gone so far as to urge members of the caucus to reach out to the White House and clue the administration into their concerns before the nomination became official, CNN reports.

Some Republicans had suggested, however, that the sexual misconduct allegations Cain faced might not be enough to sink his nomination.

“I think sexual misconduct is wrong,” said Sen. Chuck Grassley (R-IA), while adding, “if it’s a barrier to people being in public office, the president wouldn’t be president.” Sen. David Perdue (R-GA) also argued that Cain’s sexual misconduct accusations had already been “litigated” and didn’t pose a problem for his consideration of the executive.

Republicans have put up with a lot of troubling nominees. Cain could be one step too far.

Senate Republicans have helped usher through Cabinet and judicial nominees who’ve been accused of sexual assault, advocated the rollback of voting rights and questioned the role humans play in climate change.

Cain, however, appears to have pushed the limits of Republicans’ willingness to stand by the president’s selections — particularly among those lawmakers who are already less likely to fall in line.

Sen. Mitt Romney (R-UT) was among one of the earliest Republicans to come out against Cain, saying he worries Cain’s presence would make the Fed a more partisan body, given the former executive’s longstanding political support for Trump. Romney argued earlier this month that Cain would likely help Trump fulfill his plans to slash interest rates and harm the independence of the institution guiding America’s monetary policy.

“I would like to see nominees that are economists first and not partisans. I think it’s important that the Fed be a nonpartisan entity,” Romney told Politico in an interview. “The key is that someone is outside of the political world and is an economic leader not a partisan leader.”

When asked about Cain and Moore’s potential nominations, Sen. Joni Ernst (R-IA) simply laughed. “I hope the president will consider these nominations very carefully,” she told Vox. I’m not very enthused right now.”

Growing blowback against Cain’s nomination has led Senate Republicans to stage a behind-the-scenes effort to prevent his selection from being formalized by the White House, Politico reports.

“It’s hard for me to imagine that he would be confirmed,” Cramer had said.

Still, there were lawmakers within the Republican Party who remained bullish on Cain’s chances in early April. Supporting Cain could also have been a means to align themselves more closely with Trump, especially ahead of 2020.

“I think he’d probably be confirmed,” Sen. John Cornyn (R-TX) previously said.

But Cornyn, a top Senate Republican who recently pushed back at Trump’s staffing changes at the Department of Homeland Security, added that he’d like to see more consultation from the White House on nominees down the road.

“I don’t think it’s a given that everybody whose name gets floated, without vetting and without consultation, could get confirmed,” he said.

Republican seem less worried about Stephen Moore

Moore, the second person Trump has said he intends to appoint to the Fed, appears to be getting a warmer reception from Senate Republicans, even though he also faces challenges of his own.

Democrats also view Moore, a current fellow at conservative think tank the Heritage Foundation, as a baldly political choice — and pointedly wonder whether someone who’s encountered personal finance issues in the past is qualified to help run the US’s central bank.

As Amanda Sakuma wrote for Vox, Moore was previously held in contempt of court for failing to pay child support and alimony to his ex-wife in the wake of their divorce settlement. He also owes more than $75,000 in taxes to the IRS, which he says he’s been paying back in the aftermath of what he claimed was a paperwork-related mishap, according to the Guardian.

Those issues, however, aren’t necessarily disqualifying for all Senate Republicans; multiple lawmakers mentioned that they were familiar with Moore and spoke positively of his consideration.

“I know Stephen Moore, he’s a smart man, he’s the head of Club for Growth,” said Sen. Richard Shelby (R-AL). “We have to do this in regular order, but I think he would probably be a good voice on the Fed. One, he’s got to be nominated first. Second, he’s got to be confirmed.”

“I said, ‘Pay your taxes; pay your support!’” Shelby said, when asked about Moore’s financial problems.

“Stephen is a solid guy, I know Stephen pretty well. I think Stephen has been right about a few things, with regard to the Fed’s treatment of interest rates, especially,” Cramer said. “I want to hear more about the specific issues surrounding some of his financial situations.”

The North Dakota senator added that he did not see Moore’s run-in with the IRS or the divorce settlement being disqualifying for his nomination if he had cleared up both issues.

Some Republicans had said they’d be open to vetting both Moore and Cain further as part of the confirmation process, and making a decision after more steps had been taken, one way to diplomatically dodge the question of where they actually stand on the latest of Trump’s controversial nominees.

“They’re unconventional picks, and I want to see what the banking committee hearings reveal,” said Sen. Susan Collins (R-ME). “I’m sure there will be many, many questions about both nominees.”