Paul Ryan on Wednesday made a point to say he was retiring at the end of his term, not resigning, and he vowed to stay on as House Speaker through November's midterms.

Noble as it is for Ryan not to run off, rank and file House Republicans likely will force his hand to step down much sooner than that, Axios reports.

"He will be gone by the end of July," one House Republican told Axios.

With Republican incumbents - those who haven't retired - already facing strong headwinds heading into the midterms, the last thing the caucus needs is a six-month race to replace Ryan. Further, Ryan staying on ensures Republicans will be treading water until November, Axios reports.

"Scuttlebutt is that Paul will have to step down from speakership soon," a source close to leadership told Axios. "Members won’t follow a lame duck, he’ll have no leverage to cut deals, and the last thing they need in this environment is six months of palace intrigue and everyone stabbing everyone else in the back."

Majority Leader Kevin McCarthy, R-Calif., is seen as the odds-on favorite to succeed Ryan, but his inability to win over the Freedom Caucus in 2015 is what led to Ryan being tapped in the first place.

That's why some - including John Gizzi of Newsmax - think Majority whip Steve Scalise, R-Louisiana, is next in line after Ryan.

However, Scalise has said he has interest, but he will not oppose McCarthy, and that he'd run for it if McCarthy didn't or if he fell short again.

Either way, part of the issue also is what exactly either Republican is in line for - Speaker or minority leader.