(CNN) House Speaker Paul Ryan implored his conference to come together, drop internal feuds and take a breath in a closed-door meeting of House Republican members Tuesday.

"The message was for everyone to take a deep breath and not point fingers and hopefully stay on the same page," said Rep. Ryan Costello, a Republican from Pennsylvania.

Just days after the GOP failed to pass its farm bill and amid a moderate uprising on immigration, Ryan reminded the conference, according to members in the room, that they were the majority and they needed to act accordingly.

"I just heard the speaker talking about uniting," said Rep. Steve Womack, a Republican from Arkansas. "He's right. We've got to recognize that when you have the majority, act like it. It's an inconvenient truth right now. We acted like a minority party last week."

Ryan repeated that message to reporters following the conference meeting.

"Obviously last Friday was regretful. Obviously, we did not want to see our members take down the farm bill," Ryan said acknowledging that at times the system is "messy."

Ryan's message comes as moderates in the House are few as five votes away from forcing a vote on an package of immigration bills using a procedural maneuver known as a discharge petition.

It's an effort that leaders for weeks have been trying to dissuade members from supporting and a threat that led conservatives Friday to block a farm bill as leverage to get their own vote on a conservative immigration proposal. Rep. Jeff Denham, a Republican from California who's pushing the petition, told reporters that he expected there would be more signatures Tuesday, as moderate aides warn that the faction has grown frustrated with leadership's lack of action.

Meanwhile, members said that Ryan was clearly frustrated in conference with the petition and has continued to try and stave off more signatures by assuring members there will be an immigration vote the third week of June.

"I mean it's Paul Ryan. He's not a salty guy. I think he said 'gee wiz' and 'gosh' and he used the word 'crap' once," Rep. Mark Amodei, a Republican from Nevada said. "For Paul Ryan, 'crap' is pretty blue language."

"He never yells," Rep. Bill Flores, a Republican from Texas recalled about Ryan's posture during conference. "He did say 'crap.'"

In the backdrop of the immigration fight is the reality that Ryan is retiring, a fact that has some whispering -- hardly a critical mass -- that Ryan is losing his ability to control a politically divided conference. Members largely dismissed rumors that Ryan needs to go as just hearsay, but one member said it did come up.

At the news conference, Ryan attempted to brush aside questions about his leadership and whether he can remain in the job as speaker.

"Obviously I serve at the pleasure of the members -- those are the people who drafted me in this job in the first place," Ryan said on Capitol Hill today, "But I think we all agree the best thing for us is to complete our agenda and not wedge into the middle of our agenda divisive leadership elections."

Rep. Tom MacArthur of New Jersey confirmed that some of the reignited drama over leadership issues, such as whether Ryan should leave early, came up during the meeting, but he declined to give any details. MacArthur argued that Ryan hasn't lost credibility and described rumors about his esteem among the conference as merely "chatter."

On whether he thinks Ryan will stay to finish his term, MacArthur said, "I expect he will, but I'm no prophet."

Immigration -- perhaps more than any other issue the GOP faces -- has long divided the conference and exposed internal schisms. Majority Deputy Whip Patrick McHenry of North Carolina said that it's an issue that has long been tough to unite around.

Pushed on whether a few procedural tactics could be used to finally get an outcome, McHenry said "there is no fairy dust in Congress, therefore we have no immigration bill."

House Majority Leader Kevin McCarthy disputed the notion, from a report over the weekend, that he is working with the White House to get Ryan out early.

"I want to be very clear, I read that report. That report is not true," McCarthy said, with Ryan standing right beside him.