(CNN) The rumors of House Speaker Paul Ryan's looming demise are exaggerated -- at least according to the very members who would be responsible for throwing him out earlier than his planned January departure.

After a week that brought the always-divisive intra-party battle over immigration back to the forefront, which led to the failure of the farm bill , which was followed by reports of behind-the-scenes plotting for Ryan's early exit, members across the ideological spectrum were quick to defend Ryan -- at least publicly.

Even some of the most conservative lawmakers argue that Ryan's earned his right to stay.

"This 'Throw Paul Ryan overboard,' it doesn't make any sense to me," said Republican Study Committee Chairman Mark Walker, a conservative North Carolina Republican. "Yes, the longer you're here, it does become more challenging to continue to carry out what you believe should be the conference agenda, but at the same time, that's not the only thing we measure people by or I don't measure leaders by. Paul's integrity, his character's intact so I think he's earned the right to carry on the way he wants to."

For Ryan, a difficult few days belied the reality that he is, in fact, a lame duck speaker scheduled by his own decision to depart Congress at the end of his term , and he is in charge of an unruly and unpredictable conference that could muster the votes to hasten that exit.

It's also true that, contrary to all public claims otherwise, there is movement, positioning and strategizing behind the scenes about who will eventually replace him, multiple sources tell CNN. But for the moment, the actual pathway -- and perhaps more importantly, the critical mass inside the conference to get that done -- doesn't exist, lawmakers and aides acknowledged.

That doesn't mean things can't or won't shift -- Ryan is heading toward a treacherous month in June, when leaders plan to bring immigration to the floor in some capacity.

Few, if any, issues animate the conference more, particularly its always restive right flank, and a misstep could create a problem. But for the moment, members of the conference appear to prefer the status quo to the unknowns of a divisive leadership fight shortly before the midterm elections.

"He's the best spokesperson we have," said Rep. Ryan Costello, a Republican from Pennsylvania who is also retiring at the end of this Congress. "The lights never shine too bright for him having run for VP. I think he's well-regarded within the conference, he's got a good relationship in working with the Senate. Why reinvent the wheel with only six months left to go?"

Ryan tried to quell any rumors Tuesday by telling reporters that "obviously, I serve at the pleasure of the members -- those are the people who drafted me in this job in the first place, but I think we all agree the best thing for us is to complete our agenda and not wedge into the middle of our agenda divisive leadership elections."

Members not convinced Ryan leaving early would quash party infighting

Even those members who aren't fans of Ryan and have made no secret of their dissatisfaction with his leadership aren't willing to push him out now as the conference prepares to muddle its way through a messy immigration fight.

"I'm a little bit jaded. Like we worked really hard to get rid of (former House Speaker John) Boehner and then we got Paul Ryan and he was no better, so I'm not sure getting a new speaker solves anything now," Rep. Thomas Massie, a Republican from Kentucky, told CNN.

Rep. Justin Amash, a member of the House Freedom Caucus, said talk of Ryan needing to go is happening at some level, but that he's not among those calling for his ouster.

"He's disrupting things by staying," the Michigan Republican said. "He's disrupting things by leaving. I don't think there's any easy solutions, so probably the least disruptive thing is for him to just stay."

Part of the issue if Ryan left is that the turmoil in the conference, the war between moderates and conservatives and the likely futile outcome of any immigration debate would be political poison to anyone vying for a leadership role. If Ryan leaves, the problems of the conference aren't instantly resolved, they only become the next leader's battles.

Members across the conference also recognize that Ryan's ability to fundraise and campaign is still the strongest of any leader they have . Beyond his personal fundraising prowess, which shows up regularly in Federal Election Commission filings, the super PAC aligned with his operation, the Congressional Leadership Fund, has already raised more than $70 million to aid House Republicans in a year all acknowledge contains significant headwinds.

Ryan's looming departure creates some degree of a vacuum , several members told CNN -- one exacerbated by a conference that at times appears to exist to give its leaders fits. The grumbling as the immigration issue has threatened to once again envelop the conference is real -- if limited to a smaller group of members than their voices and background quotes would lead some to believe.

"I think there are members who love the idea of chaos all the time because then they become more relevant," said Rep. Tom Rooney, a Republican from Florida.

'Plan beats no plan'

The fight for what comes next has sparked behind-the-scenes conversations from Ryan's top deputies, their allies, those on K Street and in the White House about the best strategy for what comes next, sources involved in some of those discussions say.

There's also a brass tacks issue nobody seems to have an answer for: If it's time for Ryan to go, how would it actually happen? As of now, there's no clear sign McCarthy -- or anyone else -- has locked in the 218 votes needed to succeed the Wisconsin Republican and the procedural motion to bounce Ryan from his post -- a motion to vacate -- is unlikely to have the support.

"Plan beats no plan," one GOP lobbyist with close ties to leadership said. "And right now, there's no concrete plan that I've seen that moves him out."

The White House, for its part, has also stayed out of things, even as administration officials acknowledge the topic of what -- or who -- comes next has come up, but the President -- who aides on both ends of Pennsylvania Avenue say has developed a good working relationship with Ryan -- hasn't weighed in. White House press secretary Sarah Sanders said that at this point the White House is staying out of it entirely (though her decision to not outwardly back Ryan's position was noted by several GOP aides Tuesday after the daily briefing).

"At this point, that's something for Speaker Ryan and members of Congress to make that determination, not something that the White House has weighed into at this point," Sanders told reporters.

The biggest challenge for Ryan now may be keeping the discontentment of a few from ballooning into self-fulfilling prophecy.

"Once a rumor starts in this particular institution, it seems to mushroom into reality at times. I do think there has been a great deal of discussion about it, but I don't think it's been generated from him or even his thoughts," said Rep. Dennis Ross, a Republican from Florida. "We got to see how we get through this farm bill and DACA issue between now and the end of June. ... I'd like to see the speaker go out at the end of his term, but also go out with some wins such as a DACA bill and a farm bill and some appropriations bills."

But at least for the moment, even with his looming departure, a move to oust the speaker doesn't appear imminent, lawmakers said.

"Under the theory being espoused by whoever is espousing it then there are six or seven lame duck committee persons. Should we all leave too?" said Rep. Trey Gowdy, a Republican from South Carolina who is retiring at the end of the year and is a close Ryan ally. "He is uniquely well suited to be the speaker of the House and I hope he stays until the last second of the last minute, and I'm also very good friends with Kevin McCarthy, who has never once ever said a word to the contrary."