(CNN) At least two Republican House members confronted Speaker Paul Ryan on Friday morning at their caucus meeting over the firing of the House chaplain, multiple Republicans coming out of the meeting Friday told CNN.

One of the members who brought up concerns was New York Republican Rep. Pete King, who said afterward that Ryan's explanation for asking the Rev. Pat Conroy, a Jesuit priest, to resign was "unsatisfactory." Both King and Ryan are Catholic.

"To me it was an unsatisfactory answer," King said. "It is such an unprecedented action to be to only be taken for very, very serious issues. And the speaker said it was just because certain people said he was not complying with their request or was not giving good counsel. I never heard that from anyone. Anyone who I know who deals with him has the highest regard for him."

Jesuits close to Conroy told CNN that Ryan's office asked Conroy to come by the speaker's office around the Easter recess. Conroy was told he was being asked to resign but not told why. It was kind of a "thanks for your service, Padre" kind of conversation, Jesuits close to Conroy described to CNN.

The sources said Conroy was particularly perplexed because he has made strong friendships with Republicans in his seven years as chaplain and genuinely does try to stay out of politics.

Just a few days before he was asked to resign , he told Catholics in California that he is "in the one job in the US where I absolutely have to abstain from politics!"

Conroy resigned April 15 after he spent nearly seven years praying at the outset of House sessions. He wrote two versions of his resignation letters, which were both obtained by CNN by a congressional source. In the first letter, he writes that Ryan should consult with his chief of staff on a resignation date, but the second letter stated his last day would be May 24.

Ryan's reasons

It's not immediately clear why Conroy was asked to resign. CNN reported Thursday that Ryan requested the House chaplain resign from his post earlier this month, and several prominent Democratic lawmakers want to know more information as to why the Jesuit priest was asked to step down.

On Thursday, Ryan's spokesperson AshLee Strong denied that Conroy was pushed out for anything he said or did, but she didn't elaborate on what triggered the resignation request.

"The speaker consulted with the minority leader, but the decision was his," she told CNN in a statement. "He remains grateful for Father Conroy's service."

According to a person who attended Friday's conference meeting, the speaker explained that he asked Conroy to resign "based on member feedback about pastoral care."

A separate GOP member who attended the meeting said Ryan "didn't say this as bluntly but the reason for the change is that many of us like Father Conroy but we feel like he didn't do anything. We never see him. We never hear from him. We'd like to have a more active Priest/Pastor."

At the meeting, King asked Ryan if the decision was made due to any political pressure or political motivation. Ryan responded no and disputed the notion that it was related to the prayer in October on tax reform.

"Paul said that was not the issue," King said. "Paul said it wasn't solely because he want giving good service to the members. Again, I had never heard that before."

Republicans praise fired chaplain and don't know why he was fired

When asked what reasons Ryan gave in the caucus meeting for why the chaplain was fired, GOP Rep. Doug Lamborn was at a total loss for words.

"I was -- I'm not the best one to answer that because I was talking to some people on the side about personal issue during that moment that we were talking about -- but I agree with it," he responded, after pausing for five minutes searching for words.

King lamented that at the end of the day Ryan did not give "real specific issues" that led to his firing -- beyond saying there were some "complaints from some of the members about lack of the chaplain's service."

Rep. Mark Sanford, who was in the meeting and has been selected to help choose the new chaplain, told CNN that Ryan said it was "based on the input of some members" who had not been satisfied with the chaplain -- but did not or would elaborate.

GOP Rep. Mark Walker refuted rumors circulating among House members that he told Ryan he wanted a minister with a family to take over as House chaplain.

"I have no problem, Catholic, bishop, priest, male or female, my only point was that they would have some familial experience or family-based ministry," Walker told CNN. "What they thought I was talking about was somebody that had to be married -- not at all. We would want somebody who had some experience dealing with issues and families."

When asked what prompted his remarks, he said he never complained about the chaplain.

"I never even had any interaction with him ... I never complained to Paul," he said. "I know there were multiple members who did."

GOP Rep. Tom Rooney, a Catholic, told CNN before Friday's votes that he "loved" the Chaplain and that he was "a great priest." He voted "present" for the motion to investigate the chaplain firing.

"I think he is a great priest," he said. "Obviously, the Catholic Church and the Republican Party don't always marry up and neither does the Catholic Church and the Democratic Party so I don't think it's supposed to be a litmus test on whether or not it fits neatly into one party or another."

Democrats want answers

New York Democratic Rep. Joseph Crowley offered a privileged resolution on the House floor, trying to force investigation about firing of the chaplain. It was eventually voted down by the House.

Democratic Rep. Gerry Connolly, who is Catholic, told CNN Thursday he wasn't sure why Ryan made the decision and wants to the Wisconsin Republican to answer questions over "this breach that he created." Connoly's office was among several circulating a letter Friday afternoon signed by 148 members "requesting information on his decision to remove the House chaplain."

An aide for House Minority Leader Nancy Pelosi told CNN she made it clear to Ryan that she had only received positive comments about Conroy's service from House lawmakers. The aide also said Pelosi, who is also Catholic, made it clear to Ryan that she disagreed with his decision.

Pelosi released a statement Friday saying Conroy "ministered to the needs" of members and that his term had been a "blessing of grace and dignity."

"It is truly sad that he made this decision, and it is especially bewildering that he did so only a matter of months before the end of his term," Pelosi said. "The Speaker did this knowing that he had no power to fire Father Conroy and instead chose to force him out by demanding his resignation."

Rep. Emanuel Cleaver, a Methodist pastor for more than 30 years, went straight to the speaker's office on Thursday and told his chief of staff that Ryan's move to fire the House chaplain was "ugly" and "awful."

Cleaver delivered that message after he met with Conroy to hear about why he was told he needed to go and was told that Jonathan Burks, the top aide to the speaker asked him to step down and gave him the letter to sign.

The Missouri Democrat tried to meet with Ryan personally but ended up having a 30 minute meeting with Burks, who he said was "cordial and listened."

Cleaver noted that the Jesuit's career and reputation is forever damaged by the episode, "especially when a Catholic priest dismissed -- oh jeez, all this other stuff that's been going on with these priests, and this is awful, this is absolutely awful."

"I am extremely pained by the new ways that we can tell the public that we are dysfunctional," an emotional Cleaver told reporters off the House floor on Friday.

This story has been updated.