The tragedies in Dayton and El Paso occurred against the backdrop of continued frustration with what seems to be a growing frequency of mass shootings in the United States. But are they really growing in frequency? The answer to this question seems like it should be a straightforward matter of fact versus fiction: let's just count the number of times these things happen every year, throw them on a graph, and see what we find. Right? Wrong.

First, however, let me just say this article will not review explanations for why mass shootings happen. Although that is an extremely important question, it presumes we know what a "mass shooting" is in the first place. Even though this may seem obvious, my point here will be to show that we actually have no (collectively agreed upon) idea what a mass shooting is, and this is a big problem.

This is a problem because our explanations for the occurrence of something depend on our ability to distinguish that thing from other things that are different in kind. For example, if you define apples as red fruit, and I define them as green fruit, and then we individually set about counting red and green fruit, should we be surprised that we come to very different conclusions about how common "apples" are? This is not too different from the state of research on mass shootings. What follows is not an exhaustive review of the literature on mass shootings, but a brief window into how a series of data measurement challenges can undermine what we think we know about the state of the world.

A recent article by Christopher Ferguson seems to usefully attack several myths about mass shootings by simply comparing the content of these claims with the actual data. Early in the article, his myth-busting makes use of a well-known and respected dataset on mass shootings that is maintained by Mother Jones.

However, when Ferguson comes to the question of whether mass shootings are on the rise (the "myth" he says, is that they are on the rise), he quietly abandons the Mother Jones data and switches to another well-known dataset—one that is maintained in coordination among USA Today, the Associated Press, and Northeastern University. This switch in datasets would not in itself seem so strange were it not for the fact that the Mother Jones dataset contains data that directly address the question of whether mass shootings are on the rise. So why wouldn't Ferguson just stick with the Mother Jones data?

For whatever reason (he doesn't say, and I won't speculate), Ferguson takes the USA Today data and reports that the frequency of mass shooting incidents has not in fact increased between 2006 and 2019. This, he says, is true given "standard definitions" of mass shootings and according to "most data." The graph he produces is below. And out of curiosity, I created the same graph but with the Mother Jones data just to see how they compare. It is also below.

Source: Christopher Ferguson / The Conversation

Source: Mother Jones / Mass Shootings Database

Two things to notice about the two graphs. First, many more events occur within each year in the USA Today dataset than in the Mother Jones dataset. Second, Mother Jones seems to record an increase in frequency, while we observe no change in frequency since 2006 in the USA Today data. That's a pretty meaningful difference! One dataset seems to reveal that mass shootings are on the rise, and another dataset seems to show the situation is pretty much unchanged.

Remember that Ferguson tells us that his conclusions are based on "standard definitions" and "most data." So let's check that too. Ferguson cites a study by Fox and DeLateur that is critical of the Mother Jones data. According to Fox and DeLateur, the criteria that Mother Jones uses to determine whether to record an incident as a mass shooting is "hard to defend." As Fox and DeLateur describe, in order for an event to be included in the Mother Jones dataset, the shooting must have resulted in at least 4 fatalities during a single incident in a public place, the assailant must have acted alone, and their motives or association should not be connected to armed robbery or gang activity.1 Furthermore, victims must not be directly related to the shooter—otherwise the literature labels this type of event as "familicide."

Fox and DeLateur argue that we must include "mass shootings in all forms," which they do by aggregating FBI homicide data that does not exclude cases based on the restrictive criteria used by Mother Jones, such as incident location and victim-offender relationship. The only incidents excluded are those with fewer than four victims (not including the attacker), and those with a weapon other than a firearm. Fox and DeLateur find that there have been about 20 mass shootings a year, with no increasing trend, which is similar to what Ferguson observes with the USA Today data.2

A more recent study by King and Jacobson argues that the USA Today data may in fact be the most complete due to its inclusion criteria being broader than Mother Jones, and because it combines FBI data with reporting data to provide a more comprehensive dataset. The broadest event categorized in this dataset is a "mass killing," which is defined as an incident that results in at least four fatalities. Period. From this very broad criteria, King and Jacobson also distinguish between incidents that involved, for example, a firearm (vs. other weapons), direct family, or criminal activity such as theft.

What do King and Jacobson find? At the broadest level of categorization (4 people killed), there is no increase in frequency since 2006. They also examine 6 sub-categories of mass killings: 3 methods of killing (shooting, stabbing, arson) and 3 types of killing (family, public, robbery). The researchers find no increases in frequency for any of the 6 types of mass killings. The good news? Unlike Fox and DeLateur who relied on a visual inspection of the trend (i.e. do the bars in the graph above seem to be getting bigger?), King and Jacobson applied statistical techniques to determine whether an upward trend could be detected, which is a more convincing argument that mass killings are not on the rise.

The bad news? This still can't be directly compared with popular conclusions from the Mother Jones data because although Fox and DeLateur examine mass killings involving a firearm and they examine mass killings in a public place, they do not look at mass killings involving a firearm in a public place. So while their findings are most certainly useful and convincing, they are useful only for the six categories they describe. Among many important caveats provided by the researchers, King and Jacobson usefully note—contra Ferguson's claims about consensus—that "there is not unanimous agreement on an appropriate definition of a mass killing" and therefore that altering definitional criteria could lead to different results.

For example, Amy Cohen and her colleagues, provide a compelling defense of the Mother Jones data, arguing, among other things, that the inclusion criteria are indeed very specific, but this is a virtue, not a vice. Recall that Mother Jones defines a mass shooting as one that occurs in a public space, results in at least 4 fatalities, and none of the fatalities are related to the shooter. Notice that this definition cuts across 3 of the 6 categories defined by King and Jacobson—shootings, non-family, public location. Cohen et al. show that, when defined this way, a statistically significant shift in the trend of mass shootings occurs around 2011. In the three decades leading up to 2011, the researchers find that "mass shootings occurred every 200 days on average. In the subsequent three-year phase, mass shootings occurred every 64 days on average."

Unfortunately, the situation is still more complicated than restrictive criteria indicating an increase in frequency and broader criteria indicating no change. Grant Duwe uses criteria that are very similar to the Mother Jones dataset, but still finds little change in the frequency of mass shootings over time. Duwe's methodology is to start with the FBI data, supplement it with his own research data, and to apply relatively restrictive definitional criteria similar to the Mother Jones dataset. This is basically the best of both worlds: the comprehensiveness of the FBI data plus a conservative definition of mass shootings. The fundamental problem with the Mother Jones data, according to Duwe, is that it underreports the actual number of events that meet its own criteria, and when you include all cases that meet these criteria, the result is that there has been no change in frequency. Duwe speculates that these "sins of omission" are probably due to the fact that Mother Jones relies on news reporting as a source of data, and coverage is less accessible further back in time.

So what do we know? At the broadest level, defined in terms of at least any 4 people killed by anyone, by any weapon, for any reason, in any place, the frequency of mass killings seems to have remained relatively stable since at least 2006. This much seems relatively clear. However, I also find King and Jacobson's analysis convincing, which shows that some categories of more narrowly defined mass killings have also remained relatively stable. Furthermore, these analyses do not seem to contradict Cohen and colleagues, who persuasively demonstrate that, when using restrictive criteria that I think usefully align with the common notions of "mass shootings," we see that at least one dramatic form of mass killing has been on the rise. Nevertheless, Duwe throws cold water on even these findings, which again seems to suggest that the weight of the evidence is against an increasing frequency of mass shootings.

Does it matter whether mass shootings are on the rise? I think it does, but a stable rate is almost as bad as an increasing one. I mean, what's worse? The steady increase of previously rare shootings (Mother Jones) or unchanged levels of relatively common killings (FBI; USA Today)? Either way, what is revealed is that part of the challenge is to speak the same language regarding the underlying nature of the problem. Defining "mass shooting" seems like a simple matter, but it sits at the complicated intersection of many questions: Who did it? Where did they do it? Who they did it to? Why did they do it?

The risk is to cherry pick the data based on definitions of the problem that unproblematically lead to our pre-chosen favored policies. An additional problem is that a puzzle such as this requires great scientific nuance, but it is precisely in these situations when we are most likely to throw science out the window and rely on moral guidance. This isn't necessarily a vice; to proceed without moral compass on this question seems both misguided and naive. The challenge is to proceed with both heavy heart and open eyes. Toward that end, I have offered no answers, but rather some thoughts meant merely to productively complicate a picture that I think many of us on all sides believe to be all too simple.

1. In accordance with a government mandate in 2013, the Mother Jones dataset includes incidents with 3 or more fatalities, instead of 4 or more.

2. Research by James Alan Fox and his colleagues has also provided the foundation for the USA Today database. As of August 2019, the database is not yet available to the public.