Gary Kleck is the David J. Bordua professor of criminology and criminal justice at Florida State University.

It’s deja vu all over again. In a recent Politico Magazine article, Evan DeFilippis and Devin Hughes resuscitate criticisms of a survey on defensive gun use that I conducted with my colleague Marc Gertz way back in 1993—the National Self-Defense Survey (NSDS). The authors repeat, item for item, speculative criticisms floated by a man named David Hemenway in 1997 and repeated endlessly since. The conclusion these critics drew is that our survey grossly overestimated the frequency of defensive gun use (DGU), a situation in which a crime victim uses a gun to threaten or attack the offender in self-defense. But what DeFilippis and Hughes carefully withheld from readers is the fact that I and my colleague have refuted every one of Hemenway’s dubious claims, and those by other critics of the NSDS, first in 1997, and again, even more extensively, in 1998 and 2001. Skeptical readers can check for themselves if we failed to refute them—the 1998 version is publicly available here. More seriously motivated readers could acquire a copy of Armed, a 2001 book by Don Kates and me, and read chapter six.

If DeFilippis and Hughes could refute any of our rebuttals, that would be news worth attending to. They do not, however, identify any problems with our refutations, such as errors in our logic, or superior evidence that contradicts any of our rebuttals. Instead, they just pretend they are not aware of the rebuttals, even though our first systematic dismantling of Hemenway’s speculations was published in the exact same issue of the journal that published Hemenway's 1997 critique, on the pages immediately following the Hemenway article.


The authors, a couple of Oklahoma investment counselors with no graduate degrees, do not claim to have had any training in survey research methods. Like Hemenway (who is also untrained in survey methods), they believe that it’s perfectly plausible that surveys generate enormous over-estimates of crime-related experiences, as if this were the most commonplace thing in the world. The reality that survey experts are familiar with, however, is that surveys of the general public simply do not overestimate crime-related experiences.

In order for a survey respondent to report a typical DGU, she or he must be willing to report all three of the following elements of the event: (1) a crime victimization experience, (2) his or her possession of a gun, and (3) his or her own commission of a crime. The last element is relevant because most DGUs occur away from the user’s home, and only about 1 percent of the population in 1993, when we conducted our survey, had a permit that allowed them to legally carry a gun through public spaces. Thus, although survey-reported defensive gun uses themselves rarely involve criminal behavior (that is, the defender did not use the gun to commit a criminal assault or other offense), most (at least back in 1993) involved unlawful possession of a gun in a public place by the defender.

So what does research on the flaws in surveys of crime-related behaviors tell us? It consistently indicates that survey respondents underreport (1) crime victimization experiences, (2) gun ownership and (3) their own illegal behavior. While it is true that a few respondents overstate their crime-related experiences, they are greatly outnumbered by those who understate them, i.e. those who falsely deny having the experience when in fact they did. In sum, research tells us that surveys underestimate the frequency of crime victimizations, gun possession and self-reported illegal behavior. Yet DeFilippis and Hughes somehow manage to conclude that defensive gun uses—incidents that always involve the first two of those elements, and usually the third as well—are overestimated in surveys.

Like Hemenway, DeFilippis and Hughes fail to understand the most fundamental logical issue regarding whether surveys under or overestimate the frequency of defensive gun use. The point at issue is not whether there are “false positive” responses, i.e. respondents saying “yes, they used their gun defensively” when the correct answer was “no.” No one has ever disputed that there are some false positives in these surveys. But this by itself can tell us nothing about whether DGU estimates are too high or too low overall. Even if false positives were numerous, false negatives (when a respondent falsely denies a DGU that actually occurred) could be (and, according to extensive research, are) even more common. In that case, survey estimates of DGU frequency would be too low, not the enormous overestimate that DeFilippis and Hughes believe in. Since neither of those authors nor Hemenway—nor any other critics for that matter—have ever made the slightest effort to estimate the number of false negatives, they cannot possibly know whether false positives outnumber false negatives and therefore have no logical foundation whatsoever for their claims that erroneous responses to DGU questions result in an overestimate of DGU frequency.

The authors’ discussion of possible flaws in survey estimates of DGU frequency is conspicuously one-sided, addressing only supposed flaws that could make the estimates too high—but none that could make the estimates too low. As mentioned above, they say nothing about the well-documented failure of many survey respondents to report criminal victimization, gun ownership or their own crimes. Likewise, they do not mention that our estimates did not include any DGUs by adolescent crime victims, even though adolescents are more likely to be crime victims than adults, and just as likely to carry guns, albeit illegally.

To summarize, notwithstanding DeFilippis and Hughes’ one-sided cherry-picking of the research evidence, surveys do not overestimate the number of DGUs (or anything else crime-related), and at least 18 national surveys have consistently confirmed that DGUs are very common, probably more common than criminal uses of guns.

As to DeFilippis and Hughes’ motives for working so long and hard to get the DGU estimate down, I believe the most likely explanation is that they hope that total gun prohibition will one day be politically achievable, and they recognize that high numbers of DGUs each year would present an enormous obstacle to persuading Americans that disarming noncriminals would be without serious costs. No one who supported only moderate controls but who opposed total prohibition would care about high estimates of DGUs by noncriminals, since they would be unaffected by moderate controls that do not disarm noncriminals, such as background checks.

If DeFilippis and Hughes do indeed hope to see gun prohibition someday, perhaps they should be more honest with their readers as to their motives, forthrightly embracing the prohibitionist position. On the other hand, if they are not trying to advance the cause of prohibition, what could possibly justify a 2000-plus word screed pushing the long-discredited claim that Americans rarely use guns for self-protection?

Since DeFillipis and Hughes have not offered any new criticisms beyond those that Hemenway peddled back in 1997, I can do no better than to repeat the conclusions of the first refutation that I and my colleague Marc Gertz published in the Journal of Criminal Law and Criminology in 1997. They are especially noteworthy for our remarkably accurate prophecy regarding how Hemenway’s claims would be exploited by people like DeFilippis and Hughes: “Hemenway has failed to cast even mild doubt on the accuracy of our estimates. The claim that there are huge numbers of defensive uses of guns each year in the United States has been repeatedly confirmed, and remains one of the most consistently supported assertions in the guns-violence research area. Given H’s purposes, however, it is politically inconsequential that we can easily rebut all of his claims. We can be confident that ideologues will cite his series of one-sided speculations as authoritative proof that our estimates has been “discredited,” while pro-control academics who fancy themselves moderates will conclude that although maybe H was wrong on some points, he has nevertheless somehow “cast doubt” on the estimates or “raised serious questions” about them. Left unmentioned will be one simple fact: in all of H’s commentary, he does not once cite the one thing that could legitimately cast doubt on our estimates—better empirical evidence.”

***

Update: A response from Evan DeFilippis and Devin Hughes, authors of The Myth Behind Defensive Gun Ownership

When Gary Kleck can't defend, he attacks. Instead of offering new insight, Kleck instead baselessly speculates on our motives, suggesting we “hope that total gun prohibition will one day be politically achievable.” To be clear: prohibition is not something we have ever suggested in any of our writing, all of which can found at armedwithreason.com. Not that it should matter, but neither of us are merely “investment counselors” either, as Kleck suggests. In fact, DeFilippis spent most of last year helping design and analyze surveys much larger than Kleck’s.

The Florida State professor even goes so far as to describe Dr. David Hemenway, director of Harvard’s Injury Control Research Center and author of more than 130 articles and five books in Economics and Public Health (a total that includes two decisive rebuttals to Kleck and several surveys), as “a man named David Hemenway… who is also untrained in survey methods.”

Rather than confront the significant, multidisciplinary research showing that the false-positive problem is ubiquitous when measuring rare events, Kleck pretends the problem is negligible, and links us to a 1998 “rebuttal” where he references surveys that have nothing to do with rare events. As Dr. Hemenway has extensively detailed, suggesting that false negatives could somehow outweigh false positives is indulging in fantasy.

Kleck also ignores the fact that his results repeatedly fail tests of external validity. In our original article, we mention that Kleck’s data would require, impossibly, that gun owners use their gun in self-defense in more than 100 percent of burglaries. Kleck’s data also suggests that every year hundreds of thousands of criminals are shot by law-abiding citizens. But where are the hospital records to validate this claim? Kleck insists, with no medical knowledge and without citing a single study, that the vast majority of these criminals never seek hospital treatment, a claim scoffed at by medical professionals.

Kleck concludes his article by saying we “have not offered any new criticisms” and, like Dr. Hemenway before us, do “not once cite the one thing that could legitimately cast doubt on our estimates—better empirical evidence.” However, had he read the second page of our column, he would have seen that the entire point of our article was to highlight new empirical evidence debunking Kleck’s claims.

Here are the facts Kleck missed: According to his own survey more than 50 percent of respondents claim to have reported their defensive gun use to the police. This means we should find at least half of his 2.5 million annual Defensive Gun Uses (DGUs) in police reports alone. Instead, the most comprehensive nonpartisan effort to catalog police and media reports on DGUs by The Gun Violence Archive was barely able to find 1,600 in 2014. Where are the remaining 99.94 percent of Kleck’s supposed DGUs hiding?

It would be disappointing to see any professor relegated to using falsehoods and ad hominem attacks in a desperate attempt to preserve the tattered remains of his thoroughly repudiated research. Yet, such tactics are particularly deplorable when they are used in service of a gun-worshipping culture that generates tragedy on a massive scale.