The White House spent this week rolling out its task force for combating sexual violence on campus, and part of the campaign has been to routinely cite the claim that 1 in 5 female college students is sexually assaulted during her time in school. There was an immediate and frankly virulent pushback from conservative media sources like The Laura Ingraham Show and the Daily Caller, as chronicled by Media Matters, calling the White House statistics on campus sexual violence “bizarre,” “fraudulent,” and “ridiculous.”

No doubt in response to this, Glenn Kessler of the Washington Post fact-checked the stat, figuring out that the source for that 1-in-5 number is a 2007 survey commissioned by the Justice Department (during the Bush administration). Kessler finds that there’s no real problem with the survey itself, which polled more than 5,000 women from two large state universities and found that 19 percent of them experienced either rape or some other sexual assault during their time on campus. He cautions that it’s “problematic to suggest that it is representative of the experience of all college women,” but he also notes that sexual assault is a wildly underreported crime, making it difficult to get broad-based statistics that give us a clearer picture of what’s going on at every campus. (One of the White House’s initiatives is to encourage universities to collect more data on this, so the implication that the Obama administration is trying to mislead people on this front is really weird.)

Part of the “nuh-uh!” impulse coming from right-wing media might just be knee-jerk hostility to anything the Obama administration does—a phenomenon recently chronicled by my colleague Jamelle Bouie—but, to be generous, part of the problem might also be fear about what it says about college men if sexual assault is so common. Ashe Schow at the Washington Examiner griped that the report perpetuates “a culture of presuming a man is guilty,” and Robby Soave at Daily Caller complained that the report implies that campuses are “astronomically less safe than the most dangerous, crime-ridden areas of major cities.”

Let’s be clear: No one is saying that the high rates of victimization among college women mean that all men are rapists. That 1 in 5 college women have been assaulted doesn’t mean that 1 in 5 men are assailants. Far from it. A study published in 2002 by David Lisak and Paul Miller, for which they interviewed college men about their sexual histories, found that only about 6 percent of the men surveyed had attempted or successfully raped someone. While some of them only tried once, most of the rapists were repeat offenders, with each committing an average of 5.8 rapes apiece. The 6 percent of men who were rapists were generally violent men, as well. “The 120 rapists were responsible for 1,225 separate acts of interpersonal violence, including rape, battery, and child physical and sexual abuse,” the researchers write. A single rapist can leave a wake of victims, racking up the numbers rapidly, as the victim surveys are clearly showing.

This cannot be emphasized enough: The high rates of campus sexual assault are due mostly to a small percentage of men who assault multiple women. Understanding this makes the problem of sexual assault on campus much less overwhelming and, hopefully, easier to accept and address. Women aren’t running a gauntlet of would-be rapists when they go to a party or go out on dates. Most men they encounter are perfectly safe. This issue isn’t about demonizing men as a group or scaring women into thinking men are inherently dangerous. The issue here is about eradicating the small group of predators on campuses that are continually getting away with their crimes.