As speaker of the House from 2007 to 2011, Nancy Pelosi was the highest-serving woman ever to serve in American politics. Should she regain that title in January, she would make history once again. But as she fights a band of insurgent Democrats who want her replaced with a fresh face, some are wondering if gender is playing a role.

Critics include a number of Democrats who won in Republican-leaning areas after saying they wouldn’t vote for her as speaker. Her supporters argue that Pelosi was a major player behind the recent House win and an experienced legislative hand at a time when they need that.

Virginia Rep. Gerald Connolly, who is backing Pelosi, noted that it’s ironic that Pelosi would face a leadership challenge after a record-breaking election for female candidates.

“Almost everybody would agree one of the key architects of that [midterm election] victory was Nancy Pelosi,” he told reporters Wednesday. “And so the first thing we do is we thank that architect who happens to be the only woman speaker in the history of this body? We don’t even give her a gold watch, just ‘thank you for your service and get out?’ That’s an odd thing to do for a grateful caucus that has spent eight years in the wilderness.”

Seventeen current and incoming members of Congress have signed onto a letter vowing they will not back her on the floor in January, according to one Democratic source, although the official letter has yet to be released. This number is a fraction of the caucus as a whole, it presents a mathematical hurdle for Pelosi, who will need 218 votes to be elected Speaker.

Read More: Nancy Pelosi Doesn’t Care What You Think of Her

Democrats are currently projected to hold 231 seats, with six races still left to be called. That means that, right now, she currently doesn’t have the votes, and could potentially be walking a tightrope depending on the ultimate Democratic majority and who else comes out against her.

Pelosi has the weight of the party establishment behind her: Groups like NARAL and Emily’s List have endorsed her and are advocating on her behalf. She also is renowned for her ability to count votes, has been courting new and current lawmakers, and may be able to outmaneuver this mathematical hurdle should a letter actually come to fruition.

Experts who study women in politics argue that this shows a quintessential dilemma women face: they can run, and they can even win. But how far can they ascend in the governing body they serve in?

“I think this is what women who try to attain and who do attain really high level leadership positions come up against,” said Debbie Walsh, director of the Center for American Women and Politics at Rutgers’ Eagleton Institute of Politics. “It seems that women are more likely to be victimized by this than men in leadership. … [Senate Minority Leader] Chuck Schumer did not do as well with the Democratic Party in his chamber but there is no problem. He’s not held up as the symbol of all things evil.”

As Pelosi noted in her weekly press conference Thursday when faced with a barrage of questions about her bid, 14 of the 17 Democrats who have reportedly signed onto this letter are men. In addition to the three current lawmakers who are opposing her candidacy – Reps. Kathleen Rice, Marcia Fudge, and Linda Sanchez – several women elected this cycle have said they will not support Pelosi because the party needs new leadership. (Pelosi, along with her two top deputies, Reps. Steny Hoyer and Jim Clyburn, are in their late 70s and have held the top three posts in Democratic leadership for over 10 years.)

But female lawmakers who are openly opposed to Pelosi’s candidacy are frustrated by any efforts to frame opposition as a referendum on her gender, with some arguing that the reason she often touts herself as the only woman at the table is because she purposely orchestrated it that way.

“I’m a woman, and and a lot of our new members are women. And they should not be made to feel they are anti-women if they don’t want to vote for Nancy Pelosi,” New York Rep. Kathleen Rice told reporters on Wednesday. “We have an enormous number of talented women in the caucus. … I don’t think that is fair to put all the incoming women in that position where you’re telling them if you don’t support her you’re betraying your gender. Thats just not true.”

Congresswoman-elect Jennifer Wexton, who unseated Barbara Comstock in Virginia’s 10th district, was undecided who she would support heading into orientation. She was the recipient of outreach from both Pelosi and her detractors, but ultimately decided to throw her weight behind the minority leader. However, Wexton said that dissent within the caucus is in no way a factor of sexism. “I’m sure that some of the attacks on her are tinged with misogyny; ii have no doubt thats the case. But as far as within the caucus, the disagreement, I don’t believe there is a sexist element to that.”

In recent days, leaders of the anti-Pelosi faction have been floating Ohio Rep. Marcia Fudge as a possible contender to challenge her on the floor. (Fudge said Friday she would announce her intentions after Thanksgiving). Should she run — and prevail — Fudge would make history as the first African-American speaker of the House and the first African-American woman in a leadership role.

Pelosi declined to engage on Thursday when asked about the possibility of sexism being a reason for her trouble.

“I have never gone to that place,” she told reporters. “If, in fact, there is any misogyny involved in it, it is their problem, not mine.”

Get our Politics Newsletter. The headlines out of Washington never seem to slow. Subscribe to The D.C. Brief to make sense of what matters most. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Write to Alana Abramson at Alana.Abramson@time.com.