(CNN) In the days since House Speaker Paul Ryan announced his retirement , the jockeying for the speaker's gavel has centered on three names.

They are all prominent and respected lawmakers, and so far they are all white men.

At this point, no women and no minorities are in contention for the post, with most of the debate focusing largely on how House Majority Leader Kevin McCarthy of California, Majority Whip Steve Scalise of Louisiana and House Freedom Caucus co-founder Jim Jordan of Ohio have positioned themselves for such a race.

Nearly two dozen interviews with current and former lawmakers and aides revealed a Republican conference where the belief is the lack of female or minority House leaders has more to do with sheer numbers and the top positions already being occupied, but there is still a desire to expand diversity in GOP ranks. The interviews also revealed that the subject of gender and racial diversity is still taboo to discuss even among women in Congress, where the reputation of a back-slapping boys club persists.

US House Majority Leader Rep. Kevin McCarthy, a California Republican, at left, and House Majority Whip Rep. Steve Scalise, a Louisiana Republican, at right, leave after a House Republican Conference in September 2015. (Photo by Alex Wong/Getty Images)

Ryan acknowledged last week in an interview with CBS that his party needed to work harder to be inclusive, and he committed that would continue to be a focus in his post-Congress career: "We need more minorities, more women in our party. And I've been focusing on that kind of recruitment."

Many women who spoke with CNN noted that any race for leadership is hypothetical at this point. Ryan has been clear that he won't leave until January, and House Republicans must maintain the majority for there to be a speaker's race at all. But Republican women, both current and former members, who spoke with CNN confided they'd like to see the focus now be on adding more women and minorities to their ranks so when another top job opens up, there are more members to fill it.

In 2013 the House Democratic Caucus made history when it crossed the threshold of having more female and minority members than white men, but there are just 22 Republican women serving in the House, compared with 61 Democrats. The total number of minority members also lags behind in the GOP conference.

The most obvious candidate to step up for Republicans would be Cathy McMorris Rodgers, the party's conference chairwoman, who is facing her own rigorous re-election for her Washington state congressional seat, but has long shown interest in climbing the leadership ranks.

"I hope that Cathy will step up and run for speaker," former GOP Rep. Cynthia Lummis of Wyoming told CNN in a phone interview. "I hope she'll do it. I hope she runs. I say that not because I am opposed to one of the fellas being the speaker, but I think Cathy has earned the right to be among the people who are seriously discussed."

While Lummis was candid at the time of the interview that she had not spoken with McMorris Rodgers yet or encouraged her personally, her point is that the Republican conference could use more women in leadership and McMorris Rodgers, who has spent the last several years connecting with the conference, deserves to at the very least be floated.

"Sure, there aren't as many women," said Debbie Walsh, the director of the Center for American Women and Politics at Rutgers University. "There aren't as many in leadership positions, but she is the chair of the Republican conference and you would like to think in at least a list of potential speakers."

House Speaker Paul Ryan speaks to the media while flanked by Rep. Cathy McMorris Rodgers, a Washington Republican, after meeting with House Republicans on Capitol Hill in November 2015.

Only 22 women

At some level, GOP members argue it's a numbers game.

Unlike the Democratic caucus, where more than 30% -- almost a third -- of the House members are women, fewer than 10% are women in the GOP conference. That's not a large pool of applicants to draw from when it comes time to fill committee chairmanships and the leadership ranks.

Currently two women chair House committees, North Carolina's Rep. Virginia Foxx and Indiana's Rep. Susan Brooks. Tennessee's Rep. Diane Black led the House Budget Committee last year but has stepped down from her chairmanship to prepare for her run for governor in her home state.

"There aren't enough of us and we've tried to do some recruiting, but again, when you look at the farm team, whether it is women in the statehouse or city council or women mayors across the nation, many of them have to be talked into it, as opposed to just feeling like they have something to offer," said Rep. Lynn Jenkins, a Republican from Kansas who is leaving Congress next year.

Kay Granger, the only Republican woman ever to win a House seat from the state of Texas and a top candidate to lead the powerful House Appropriations Committee next year, says the women of the GOP have been talking informally in recent days about how to become more visible, but that it's up to members to stage a run, a move that takes fundraising prowess, strategic planning and long-term networking.

"There's more talk about it. There's more talk primarily among the women, frankly, (who are) of course saying some of the problems that we've watched happen ... we feel like if there were women in more prominent positions, there would be less focus on, well, what are your women doing? ... There's more awareness," Granger said.

Reps. Peter Roskam, Susan Brooks and Chairman Trey Gowdy of the House Select Committee on Benghazi speak to reporters at a news conference on the findings of former Secretary of State Hillary Clinton's personal emails on March 3, 2015.

No one wants to talk about gender and race diversity

Over and over again, members who spoke with CNN wanted to avoid attributing the lack of women or minorities in the GOP leadership pipeline to obstacles they experienced within the conference.

One Republican congresswoman told CNN that she doesn't see any obstacles in the way for women in the conference.

"This is the thing that frustrates me. We're on the same level as men. Men are not better than women," she said. "We don't believe that we're any less than the men are or that we're treated any differently. We're not."

Part of the issue for women in the GOP, some argued, is that the party isn't focused on the need to explicitly build up diversity in its ranks in the same way that Democrats have focused on it in recent years.

Then-Rep. Cynthia Lummis, a Wyoming Republican, is photographed on Capitol Hill in Washington, D.C. in March 2015.

"Let me suggest that it is harder for women in the Republican conference than it is in the Democratic conference, because Democrats at this point in their history are fixated on identity politics," Lummis said. "If you are black, Hispanic, Asian ... you are cultivated within the Democratic ranks because of their current fixation on identity politics. The Republicans are not focused on identity politics."

Rep. Carlos Curbelo, a Hispanic Republican from Florida, said diversity was "sorely needed" in his party, but when asked whether a recent tweet from President Donald Trump depicting a photo of the GOP leaders -- all white men -- was an optics problem, he pushed back.

"Look, I am not into dissing white men," he said, noting a recent incident in which House Democratic leader Nancy Pelosi of California pointed out there was a bipartisan group of five white men and no Hispanic members working on immigration. "I think that is as bigoted and as discriminatory as discarding or disqualifying minorities and women, so I'm not into that, but I do think that diversity is good, and I do think our party is due to have more minorities and women in positions of leadership."

Instead, when talking about why there aren't more women in leadership, lawmakers often cite lack of seniority, interest, raw numbers or other priorities like raising young children as reasons.

"I think the opportunity is always there, but I think it comes down to what the individual focus is of the individual woman. I don't think there are any obstacles prohibiting anybody from doing anything that they want to do in leadership, and I don't think it's a gender issue," Indiana's Rep. Jackie Walorski said. "I think if someone wants to pursue it, the doors are open to do that, but I think for me, I am on a committee that I really worked hard to get on with Ways and Means."

Mary Bono, a former Republican member from California, told CNN that when she was in Congress she couldn't have imagined pursuing a leadership role when she was balancing her family and serving in Congress.

"A lot of women, they do their day job and then they have to do their night job. Help their kids with homework ... most male members seem to have a little bit freer schedule," she said.

Bono said that even starting to build the relationships she would have needed to stage a run for speaker would have been too much.

"I had a lot to do. To throw in a leadership race would have been absolutely impossible," she said.

But men in Congress also have young children. It's not just McMorris Rodgers, the first woman to give birth three times in office, on the leadership team trying to balance family responsibilities. Scalise and Ryan have talked about their own juggling acts. Ryan, who also has three children, cited not wanting to be just a "Sunday dad" as a leading reason why he was stepping down.

Walsh -- of the Center for American Women and Politics -- argues that women and men are perceived differently when they talk about the impact that raising children has on their career decisions. For men, taking a step back from work and citing family obligations is celebrated. For women, it can become a bigger factor in deciding when to run.

Marilyn Musgrave, a former Republican congresswoman from Colorado, said her children were older when she began her service in the state Legislature.

"When I was in Congress I was so glad my husband and I were empty-nesters," she said. "It still was hard doing all the family stuff. If I had small children, I don't know how I would have done it."

GOP congresswomen: Women wait to be asked

In the early days, when McMorris Rodgers was gaining prominence in the conference, it was former House Speaker John Boehner who encouraged her to run for leadership and it was the support of prominent chairmen that helped elevate her to conference chair in a hard-fought race against then-Rep. Tom Price of Georgia.

"I just think as a general rule women feel like they need to be asked to step up in leadership, whether it is in the boardroom or in politics, and that's unfortunate, because men tend to wake up one day, look in the mirror and say, 'Darn, I'm good. I'm going to run for Congress' or 'I'm going to be speaker of the House.' I haven't met many women who think like that, so we need to change that," Jenkins said.

Rep. Mia Love, a Utah Republican, speaks during the Conservative Political Action Conference in March 2016 at National Harbor, Maryland.

Rep. Mia Love, an African-American Republican from Utah, said it took multiple people to convince her to even run for Congress and that if Republicans want more women,they need to personally make their pitches.

"I think we need to go out and tell women: You are experienced enough, you are qualified enough, you have something to offer," Love said.

Bono said that when she was in Congress she wanted to try to get her female, Republican colleagues more of a chance to confidently pursue opportunities. When she entered Congress, there was no formalized group for the small number of Republican women to regularly meet, so she started one: the Republican women's policy committee.

She initially imagined it would be a place women could get media training, but it quickly turned into more than that.

"What I then heard the most was women just wanted some moral support for the job," she said. They wanted to talk about district challenges, media challenges, family life, sick husbands, weddings and divorces.

But Bono believes that in the years after she left, the group fizzled out.

Another former Republican congresswoman, who spoke on the condition of anonymity in order to speak freely about her time in Congress, said it was hard to try to get the women together to lobby the leadership as a collective.

At one point, the former member recalled, she wanted to try to convince some of her female colleagues to organize and talk to leadership about including more women in discussions about health care in the conference. She confided in a fellow female member about her plans and the friend remarked that the men in the conference didn't like it when the women called attention to the lack of representation.

"She made the comment that one person in leadership had said we don't like it when the women said these things," the former congresswoman recalled.

Top women are leaving

Another factor in the race for leadership roles is the fact that some of the GOP's prospects to step up for leadership roles are leaving the House altogether.

Black, the former chairman of the House Budget Committee, is running for governor of Tennessee, fellow Tennessean Rep. Marsha Blackburn is running for retiring Republican Sen. Bob Corker's seat , South Dakota's Rep. Kristi Noem is running for governor and Arizona's Rep. Martha McSally is running to replace the retiring Sen. Jeff Flake of Arizona.

One GOP aide told CNN that some women may be leaving because it may be "hard for them to imagine their future here."

But the women who are leaving are reluctant to talk about why.

Blackburn waved off CNN's attempt to get her to weigh in on the current House GOP leadership race, saying, "I'm focused on other things."

When CNN pressed the Tennessee Republican, who has held senior positions on top committees, if she decided to pursue a position outside the House because of the lack of leadership opportunities for women, she curtly replied, "Nope," and quickly moved on down the hallway.

McCarthy argued that the fact so many women were pursuing jobs outside of the House was a positive sign for the women in the conference.

"I think the higher office proves what great individuals they are on serving," he said. "They're going for governors, they're going for senators, higher office. It's a harder office to win, and they all have a great opportunity to do that and they're the front-runners in those positions."

The makeup of a district

For some women and minorities in the GOP conference, the makeup of the district they're from could also make it a challenge to be associated with -- let alone be in -- leadership.

A handful of women and minorities in the GOP conference are in some of the most competitive districts the GOP has to defend every two years.

Rep. Barbara Comstock, a Republican from Virginia, is consistently cited as in a toss-up race, as is Love, who's from Utah. Rep. Will Hurd, an African-American Republican from Texas, and Curbelo, a Republican from Florida, are also in some of the toughest districts in the conference.

Former California Congresswoman Bono said her competitive district was one reason why she was more focused on putting her efforts into focusing on policy.

"For me in a swing district, it would have been hard to be on the team at times," Bono said.

At the end of the day, the women and minority members who spoke with CNN for the story said the lack of diversity wasn't about bias or anyone disregarding candidates on the basis of race or gender. It was about choice: the choice about balancing family, running for office in the first place, running for leadership or even choosing your electoral future over a leadership post. The race for speaker, they argued, was wide open.

"It's premature that there won't be any (female or minority) candidates. There's no speaker's race going on right now," Curbelo said. "You got to step forward. There's no waiting to get invited to the prom."