House Speaker Paul Ryan says he’ll remain the top Republican in the GOP-controlled chamber until retiring in January, despite calls for a fast leadership transition and concerns from within his caucus about a dragged-out process hurting the party in the November midterms.

“I don’t think anybody's election is going to hinge on if Paul Ryan is speaker of the House,” he told Fox News, after announcing Wednesday that he’s not seeking re-election this year and as the behind-the-scenes jockeying among House Republicans for his position came more into public view.

Ryan, a Wisconsin Republican, also dismissed the argument that him remaining speaker until year’s end would damage party efforts to keep its House majority as “just internal D.C. gossip.”

Texas Rep. Mike Conaway, chairman of the House Agriculture Committee, is among those in the House Republican Caucus backing Ryan and his plans to stay -- arguing a leadership election now would distract members from governing and keeping control of the lower chamber.

“Any discussion about leadership elections is entirely premature,” he said Sunday. “Our focus needs to remain on promoting policies to benefit the American people and maintaining our strong, Republican majority in the 2018 mid-term elections.

"Speaker Ryan has a proven record of leading this conference to victory. And any leadership changes at this point would only distract from the work at hand.”

But some rank-and-file House members appear concerned.

“No disrespect to Paul, but quite frankly, you want somebody who’s got skin in the game after the elections,” Rep. Mark Amodei, R-Nevada, told The Washington Post.

Amodei was perhaps the most public in his concerns, while others questioned whether Ryan could, as a lame-duck speaker, lead the chamber toward passing any meaningful legislation -- though a slim legislative agenda ahead allows incumbents to focus more on re-election as primary season heats up.

Other House Republicans argue that Ryan’s announced retirement, after roughly two decades in Congress, would diminish his ability to raise funds, which brings millions of dollars for GOP candidates.

“I’ve broken every record that any (House) speaker has ever had on fundraising,” Ryan said on the day of his retirement announcement and one day after Politico reported he'd raised $11.1 million in the first quarter of 2018, bringing his total for this year’s election cycle to $54 million.

Ryan supporters, particularly House committee chairman, backed Ryan in force this weekend, amid calls for a hastened transition.

“Congress has a huge amount of work to do over coming months, including all 12 appropriations bills,” House Appropriations Committee Chairman Rodney Frelinghuysen, R-N.J., told Fox News on Sunday. “We need strong and stable leadership to get it done, and that includes Paul heading the ship for the duration of the session.”

While Amodei sounded perhaps the most upfront about his wishes, Rep. Tom Graves, R-Ga., suggested to Politico that having “our strongest leadership in place” would yield more success in the midterms, as energized Democrats appear to have a very realistic chance of winning roughly two dozen GOP-held House seats to retake the chamber.

The party not controlling the White House historically wins about 24 seats in a midterm. And 38 House Republicans are not seeking 2018 re-election.

Amid weeks of speculation about Ryan’s departure, the contest for his post was being considered a two-man race between House Majority Leader Kevin McCarthy, the No. 2 man in GOP House leadership, and Majority Whip Steve Scalise, the No. 3 man.

However, Louisiana's Scalise has indicated he won’t challenge McCarthy if the California lawmaker seeks the speakership. And Ryan now appears to be backing McCarthy.

Conservatives appear to want the speakership vote to be held after the elections, to give new members, not outgoing ones, a vote. McCarthy’s 2015 bid to become speaker failed in part because he didn’t have the votes from the House Freedom Caucus, the chamber’s roughly 35-member conservative wing.

And at least some members and a leader of the Tea Party movement have publically and privately suggested they could add to the congressional ranks after November, giving them more power in leadership voting.