AMSTERDAM—Is being a woman a disadvantage when you're applying for grant money in the Netherlands? Yes, say the authors of a paper published by the Proceedings of the National Academy of Sciences (PNAS) this week. The study showed that women have a lower chance than men of winning early career grants from the Netherlands Organization for Scientific Research (NWO), the country's main grant agency.

NWO, which commissioned the study, accepted the results and announced several changes on Monday to rectify the problem. "NWO will devote more explicit attention to the gender awareness of reviewers in its methods and procedures," a statement said.

But several Dutch scientists who have taken a close look at the data say they see no evidence of sexism. The PNAS paper, written by Romy van der Lee and Naomi Ellemers of Leiden University's Institute of Psychology, is an example of a classic statistical trap, says statistician Casper Albers of the University of Groningen, who tore the paper apart in a blog post yesterday. (In Dutch; a shortened translation in English is here.) Albers says he plans to send the piece as a commentary to PNAS as well.

Van der Lee and Ellemers analyzed 2823 applications for NWO's Veni grants for young researchers in the years 2010, 2011, and 2012. Overall, women had a success rate of 14.9%, compared with 17.7% for men, they wrote, and that difference was statistically significant. But Albers says the difference evaporates if you look more closely at sex ratios and success rates in NWO's nine scientific disciplines. Those data, which Van der Lee and Ellemers provided in a supplement to their paper, show that women simply apply more often in fields where the chance of success is low.

For instance, women accounted for 51% and 49% of the applications in medical sciences and social sciences respectively; as it happens, those fields had the lowest overall success rates (14.9% and 13.4%, respectively) within NWO. By contrast, women made up only 12% of the applicants in physics, where the overall success rate is highest, at 26.3%. All in all, men outperformed women in five of the nine disciplines; women did better in four. But in none of the disciplines is the difference significant, Albers writes. He says the study is an example of Simpson’s Paradox, the statistical phenomenon that an aggregated set of data looks different when it is broken down.

As it happens, the most famous example of the paradox concerned academic gender bias as well. In the 1970s, the University of California, Berkeley, was sued because its graduate school admission policies seemed biased against women. But an analysis published in Science in 1975 showed that none of Berkeley's faculties were biased; women had much lower overall success rates because they applied more often at faculties that were harder to get into.

"I'm surprised that the authors, the reviewers, and the editors [of the PNAS paper] didn't see this," Albers says. "You don't have to be all that knowledgeable about statistics to spot this problem."

Ellemers—who recently moved to Utrecht University—says she and Van der Lee were fully aware that the picture looked different when broken down by discipline, and that the paper elaborates on these differences. The overall success rates "were seen as a problem by NWO," and "were the starting point for our analysis," says Ellemers, who's "not convinced" that Simpson's Paradox invalidates the findings. The overall figures aren't the most important finding anyway, she says; the paper looked at other aspects of the application process in more detail, including NWO's instruction and evaluation materials, which they found to contain "gendered" language favoring male applicants. The critics "decided to focus on one [statistical] test and burn down the entire story based on that," she says.

But several other scientists agree with Albers's conclusion that there is no evidence of sexism. "An accusation of sexism, especially if it's institutional, has to be made very cautiously," Daniel Lakens of the Eindhoven University of Technology and Rolf Hut of the Delft University of Technology wrote in a letter published in the Dutch newspaper de Volkskrant this morning. "That didn't happen here." Lakens and Hut aren’t claiming there's no sexism in science, and the profession needs more diversity, Hut emphasized in an email to ScienceInsider. "But you don't advance that good cause by accusing institutes ... of sexism based on incorrect conclusions," he writes. Even if you do accept the researchers' methodology, Albers says, they have a weak case. The difference between male and female success rates is small, and with a p-value of 0.045, the effect is barely significant. What's more, women have bested men in years not included in this analysis; for instance, in the 2015 round of Veni grants, announced in August, women had a 14.9% success rate, versus 13.9% for men. "Sometimes women do better, sometimes men, as you would expect," Albers says. "That's how chance works."

A spokesperson for NWO referred ScienceInsider's questions about the PNAS paper to the authors, but said the organization will go ahead with its reforms because they're a good idea, regardless of the paper’s quality. Among other things, NWO is planning a pilot experiment to see whether additional training for reviewers can improve women's success rates.

*Update, 24 September, 3:00 a.m.: A few sentences have been changed to better reflect Ellemers' response to the criticism.