Since House Majority Leader Kevin McCarthy (R-CA) dropped from the speaker race, Republicans have coalesced around the idea of Ways and Means Chairman Rep. Paul Ryan as their knight in shining armor, even as Ryan himself has expressed reluctance to enter the race.

However, while some of the hardliners who pushed back at McCarthy have expressed their support of Ryan, other conservatives are already begin to raise their concerns about Ryan’s record. The question is not just whether Ryan would have the votes necessary to win the speakership on the House floor, but also his ability if elected to bring the hardliners in line and avoid shutdowns, debt defaults, and the array of looming government crises.

Coming out of Friday morning’s party meeting, Rep. Louie Gohmert (R-TX) told reporters he could not back Ryan because of his support for the government bailout of Wall Street in 2008 during the depths of the financial crisis.

“I love Paul. He’s one of the smartest guys here,” Gohmert said. “Back in 2008 there were a number of us that committed that we simply could not ever support a speaker who fought so hard to pass the Wall Street bailout.”

Likewise, Rep. Thomas Massie (R-KY) withheld his support of Paul in an interview Friday with CNN and said he would stick with his previous support for Daniel Webster (R-FL) for speaker.

“Some of my conservative colleagues remember Paul Ryan’s passionate please for the TARP, the Wall Street bailout — he was asking them to vote for it several years ago,” Massie said, “I don’t have a problem with his ideology. I would want to talk to Paul Ryan about why he kicked conservatives off the budget committee, when he was chairman of the Budget Committee. So I am still supporting Daniel Webster.”

Ryan’s more moderate stance on immigration reform could also cause ire among the caucus’ hard right, according to at least one report.

Some conservatives already girding for battle to derail a potential Paul Ryan Speakership. Don’t like his stance on immigration. — Chad Pergram (@ChadPergram) October 9, 2015

One conservative mbr on Ryan who asked not to be ID’d: He’s the mouthpiece of the establishment — Chad Pergram (@ChadPergram) October 9, 2015

Some conservative mbrs wary of Ryan. 1 mbr who asked not to be id’d: He voted for TARP..He’s the go-to guy for Boehner — Chad Pergram (@ChadPergram) October 9, 2015

Resistance to Ryan is emerging outside the walls of the Capitol as well.

The Conservative Review called Ryan “the absolute worst choice for speaker,” pointing to his stances on TARP and immigration, but also 2013’s Murray-Ryan deal to temporarily suspend the budget caps, which it said, “actually countermanded the few budget victories we’ve had over the past few years.” It also accused Ryan of being a “wet blanket” on the push to defund Planned Parenthood and decried his vote in favor of a bill that would prohibit discrimination on the basis of sexual orientation.

Conservative bombthrower Erick Erickson also warned conservatives of “cult of personality” in a post titled “Paul Ryan is a Dangerous Pick for Conservatives.”

“Paul Ryan is a creature of Washington. He worked on Capitol Hill, worked in a think tank, then went back as a congressman. He speaks Washingtonese with the best of them,” Erickson wrote. “He is a team player and conservatives in Washington cannot afford to be on Team DC-GOP when the American people are mad as hell at Washington.”