I’m a physicist by training, with four years of college and six of graduate school. During that time, I had a total of three female professors in my math and science courses (one each in math and chemistry in college, one in physics in graduate school). If I’d happened to pick different classes, I might have avoided ever having a female professor in science. My situation isn’t unique. Less than a quarter of all physics professors are women, and less than a tenth of all senior physics faculty are women.

That’s why I was very surprised and skeptical when I read a recent paper by Cornell University psychologists Wendy M. Williams and Stephen J. Ceci, which posits “a surprisingly welcoming atmosphere today for female job candidates in STEM disciplines.” Williams and Ceci created imaginary job candidates for entry-level faculty positions (assistant professors, to use the jargon) who were highly qualified in every way but varied in both gender and lifestyle details, including marital status and children in their homes. When they surveyed faculty members in four disciplines, they found roughly a 2-to-1 preference for hiring the female candidates over the male candidates. Nature, the Washington Post, Inside Higher Ed, and other outlets ran largely uncritical stories about the study, while CNN published an op-ed by Williams and Ceci themselves to advertise their work.

“The low numbers of women in math-based fields of science do not result from sexist hiring,” they wrote for CNN, “but rather from women’s lower rates of choosing to enter math-based fields in the first place, due to sex differences in preferred careers and perhaps to lack of female role models and mentors.”

Unfortunately, the study contradicts every other study about the problems women face in academia—and what’s more, their own research doesn’t back up their conclusions. Sexism is an ongoing problem in universities and science, technology, engineering, and mathematics fields, with persistent bullying of female faculty, prejudice against mothers, barriers to promotion, and lower pay than male colleagues for equal work. (In fact, among all of these well-documented issues, focusing narrowly on hiring practices feels wrongheaded.)

Rather than examining actual hiring practices, Williams and Ceci based their strong conclusions on a voluntary survey sent to faculty members. They sent materials to equal numbers of male and female faculty, even though men and women are not equally represented in the departments they polled. When they analyzed their results, they seemingly did not control for the rank of the faculty respondents, which is crucial because more men hold senior-level positions and have more hiring power.

In the main part of their experiment, Williams and Ceci sent surveys to 2,090 faculty members in two math-intensive fields (engineering and economics) as well as two less math-focused disciplines (biology and psychology) where women are proportionally better-represented. (I’ll let others argue over whether economics is really a STEM field.) The surveys asked faculty members which job candidate they would hire out of a list of imaginary applicants, based on narrative essays with details about the candidates’ credentials and personal lifestyles.

Alas, that’s not how universities hire faculty members in real life. Instead, they use a curriculum vitae and short essays. Then there was the candidate pool: While Williams and Ceci included one weaker “foil” male candidate in the list, the other fake candidates were unrealistically good, with better qualifications than most on the market. And of course there’s the question of whether people’s good intentions might cause them to respond to a survey differently than they would behave in real life.

This kind of study is incredibly frustrating because it tells a story many of us would love to believe: that gender discrimination in academia is dying. Williams and Ceci rightly note that there are serious cultural biases pushing girls and women away from STEM, especially the more math-oriented fields. Those biases stymie female ambition long before women would be in a position to apply for university jobs. In physics, for instance, newly minted female job candidates are hired in rough proportion to the number of Ph.D. degrees handed out to women—stats that square with the rosy picture painted in Williams and Ceci’s paper. But women aren’t promoted to higher levels in the same proportion. According to the most recent American Institute of Physics analysis, women constituted just 18 percent of all physics and astronomy university faculty in 2010, with that proportion dropping with each higher rank. Given the power that senior faculty members have over promotion within their departments, simply hiring more women isn’t solving anything: They aren’t getting tenure, they aren’t staying, and older men still continue to control departments.

I’ve focused on physics here because that’s the field I know best. But male faculty members in plenty of fields tend to hire other men, even in areas where there’s more gender balance. Even women tend to be biased against other women in math-intensive disciplines. While the relative number of women has increased in fields like physics, it has decreased in computer science, a figure hard to reconcile with the conclusions of Williams and Ceci. And women are consistently paid less than their male counterparts— something that holds true across STEM disciplines.

Williams and Ceci also point out that men apply for jobs a lot more than women do, but even that fact doesn’t support their conclusions because men are more likely to apply for jobs for which they’re underqualified. In other words, out of a large pool of applicants—a likely case in this crappy job market—a large fraction of the male candidates can be ruled out because they were applying on spec, which skews the numbers but says little about the truly qualified applicants.

The unpleasant truth is that women face a lot more challenges in STEM than university hiring practices. Williams and Ceci cloud the issue, both by their methodology and by their conclusions, which are contradicted by other research. We need to confront biases head on if we’re to fix the problem of sexism in STEM, a problem we can’t simply explain away with surveys and op-eds.

Thanks to Bethany Brookshire, Laura Helmuth, and Shannon Palus for helpful conversations about this paper and related issues. This article owes a lot to them.