A Comparison of Two Studies

We have reviewed the study of DGU by McDowall using data from the National Crime Victimization Survey (NCVS), and the study of DGU by Kleck on the National Self-Defense Survey (NSDS). Even though both studies purport to estimate DGU is the USA, the results of the two studies are wildly inconsistent: McDowall reports there are approximately 100,000 DGU in the US every year, Kleck reports there are around 2.5 million DGU in the USA every year.

In reviewing the methods of the two studies, it is now easier to see why they came up with such different estimates of DGU. Dr. Kleck's study was designed to capture as many reports about DGU as possible:: the survey was made very “friendly” to gun owners – their anonymity was assured and they were given multiple opportunities to talk about defending themselves with a gun. The NSDS was designed to elicit reports of DGU from respondents, and it did just that.

The NCVS on the other hand, was not designed to research DGU, but only included a few questions about defending against crime as part of an effort to understand the larger phenomenon of crime victimization. And not every subject in the NCVS was asked about incidents of DGU. It should be no surprise that the NCVS estimate of DGU is smaller than that of the NSDS

Technically and methodologically, the NCVS is a superior study. The study is conducted by the US Census Bureau, a highly reputable and well-respected source for data about the US people. It has a larger sample size by a factor of 20. The sample is more representative of the US population; indeed, a study that only collects information from people who answer the home phone and agree to participate in a survey about guns is in all likelihood not representative of the US population. Perhaps because of this, Kleck neglected to include a comparison of his survey respondents with the US population on demographic variables - an omission that would not have passed peer review in any respectable medical journal. Because the NCVS survey has such a large sample and is highly representative of the US population, the study results are directly proportional to the wider US population: no weighing or unweighing of study data is needed to extrapolate results to the US population. But, the NCVS is likely to downwardly bias estimates of DGU because only crime victims are asked about using a gun defensively, not all types of crime are included in the survey, and some of the crimes that are tracked are under-reported in the survey.

At the heart of the differing estimates of DGU is the questions the two studies used to elicit responses about DGU from study subjects. The Kleck study asked all study participants if they had used a gun to protect themselves from crime, the McDowall study only asked questions about defending against crime from subject who reported being the victims of crime. In 2000, McDowall reported on the results of an experimental survey in which he asked 3000 subject BOTH the questions about defending against crime as those questions were put forth in the Kleck and McDowall studies (link: note - paywall). Interestingly, asking questions about DGU using the methods of the Kleck study yielded three times more reports of DGU than asking questions about DGU using the methods of the NCVS study. McDowall drew two conclusions from this: 1) the survey methods used account for the divergent results of the NCVS and NSDS studies; and 2) that the NCVS and the NSDS measure responses to different provocations.

In his own study report, Kleck criticizes the NCVS



“In the context of a non-anonymous survey conducted by the federal government, an R [respondent] who reports a DGU may believe that he is placing himself in serious legal jeopardy. For example, consider the issue of the location of crimes. For all but a handful of gun owners with a permit to carry a weapon in public places (under 4% of the adult population even in states like Florida, where carry permits are relatively easy to get), the mere possession of a gun in a place other than their home, place of business, or in some states, their vehicle, is a crime, often a felony. In at least ten states, it is punishable by a punitively mandatory minimum prison sentence. Yet, 88% of the violent crimes which Rs reported to NCVS interviewers in 1992 were committed away from the victim's home, i.e., in a location where it would ordinarily be a crime for the victim to even possess a gun, never mind use it defensively. Because the question about location is asked before the self-protection questions, the typical violent crime victim R has already committed himself to having been victimized in a public place before being asked what he or she did for self-protection. In short, Rs usually could not mention their defensive use of a gun without, in effect, confessing to a crime to a federal government employee.” “Even for crimes that occurred in the victim's home, such as a burglary, possession of a gun would still often be unlawful or of unknown legal status; because the R had not complied with or could not be sure he had complied with all legal requirements concerning registration of the gun's acquisition or possession, permits for purchase, licensing of home possession, storage requirements, and so on. In light of all these considerations, it may be unrealistic to assume that more than a fraction of Rs who have used a gun defensively would be willing to report it to NCVS interviewers.” (pg 156, Armed Resistance to Crime, Kleck & Gertz; 1995)

In effect, Kleck says the NCVS study under-reports DGUs because the NCVS might not include episodes of defensive gun use in which the defender owns or uses the gun illegally. However, owning or using a gun illegally is not a DGU, it is a criminal misuse of a gun. The NSDS was designed to count these illegal uses of a gun as valid and true DGUs. In sum, when Kleck reports the number 2.5 million per year, he is reporting on the combined number of DGU PLUS criminal uses of a gun. Knowing that the estimate from the NSDS counts both DGU and criminal misuse of a gun, it is clear why the NSDS has a higher estimate than the NCVS.

In 2000, David Hemenway at the Harvard School of Public Health published the results of a small telephone survey asking about DGUs. The study included having five active criminal court judges assess whether the self-reports of DGU were likely to be legal gun use or illegal gun use. For the purpose of the study, it was assumed that each study respondent who reported a DGU had a legal permit to carry a gun and honestly and accurately reported the gun use. Out of 4400 respondents, 152 reported experiencing 271 incidents where a gun was used with hostile intent against them. In the sample, 43 respondents reported using a gun to defend themselves for a total of 146 DGUs. When the criminal court judges were asked to review the reports of DGU, 51% of the DGUs were judged as “probably illegal”, and 43% of the DGUs were judged as “probably legal”. Hemenway and his research team concluded that guns are used more often to intimidate and threaten than they are used in self-defense, and that the majority of reported DGU are illegal (link).

In addition to the problems and challenges of mounting a large nation-wide survey of the behaviors of the US population, both the NCVS study and the NSDS study suffer from the inherent difficulty in determining what is and what is not a defensive gun use (see the Firearm Law and Policy diary Defenisve Gun Use: You Decide). Any study that would hope to accurately count DGU in the USA must first address this difficult quandary. This is not impossible, and thoughtful study design can help in accurately counting DGUS. Neither the NCVS nor the NSDS can claim to have done this.

For better or worse, there have been no subsequent efforts to repeat a large nation-wide studies of DGU. Nor is there likely to be any such studies because 1) such studies are expensive and time-consuming; 2) any new study would need to demonstrate significant improvement(s) in methodologies over the older studies; 3) the controversy surrounding the previous studies of DGU are likely to overshadow any new findings; 4) funding challenges in general are compounded by the congressional efforts to cut funding from the CDC for research on the topic of guns.

Summary

This concludes the Firearm Law and Policy group series on defensive gun use in America. We first reviewed the findings of the CDC report on gun violence and found the CDC could only conclude that there is considerable disagreement among empiric studies on the incidence of DGU in America (link). We then took a close look at the term "defensive gun use” and tried to define it (link). It turns out the term "defensive gun use" means different things to different people - and is highly dependent on which which way the gun is pointing. What a survey respondent might call a DGU is likely to be very different from what a researcher is hoping to count as a DGU - and both can be very different again from what a court might recognize as a DGU. We have examined the methods of two well-known empiric studies of DGU in America, the NCVS survey and the NSDS survey. The studies differ in large ways on purpose, methods, and findings. While the NCVS is a superior study in many regards, it likely undercounts DGU. On the other hand, the NSDS survey counts both DGU and criminal use of a gun to arrive at a gross overcounting of DGU.

In sum, "defensive gun use" is an ambiguous thing at best, and very difficult to estimate through empiric studies

In all likelihood, further efforts to estimate DGU through surveys and polling will suffer from similar methodological difficulties and considerable controversy. If knowing the incidence of DGU is important, it may be better to stop trying to estimate the incidence and start directly counting DGUs. This could be done by examining police records for accounts of DGU. This would be a large and tough job: either police records from all municipalities would have to be examined, or representative samples would need to be drawn up. The examination of police records would have to happen yearly if we wanted to track over-time increases and decreases in DGU. The advantages of counting DGU directly from police records are: a) the who, what and where of the incident; b) what was the criminal /offender doing? c) was an actual crime in progress or imminent? d) did the defender use a gun to defend him/herself? e) was anyone injured? f) did the defender legally own and use the gun? g) did the defender's use of a gun escalate or de-escalate the criminal activity? and perhaps most importantly, h) an objective accounting of the episode that is not dependent solely on the words of the person holding the gun. It would be very difficult if not altogether impossible to get this level of detail from a survey or self-report. Yearly examinations of police reports will provide the most accurate measurement of DGU in America.