And the mass shooter rate is fabulously low to begin with.

The media cycle this weekend has landed on a new study published by the BMJ, (archive link) on the connection between state gun laws and rate of mass shooting incidents. It’s been picked up by Vox, Yahoo, and elsewhere. Predictably, this study is being leveraged in our collective social media metabrain to push the idea that more gun laws reduce mass shootings.

The study by Paul Reeping, Bindu Kalesan, and Charles Branas, has couple of real howlers buried in it that we need to talk about, and the conclusions it draws are technically correct but misleading in practice. Let’s look at the study and then speak of what useful conclusions we can draw from it.

The Study

You can find the study, titled “State gun laws, gun ownership, and mass shootings in the US: cross sectional time series,” here. (academic citation: BMJ 2019;364:l542) An archive link is here. I encourage readers to go there, read it, and come back, but here’s a brief summary from their abstract:

Exposure An annual rating between 0 (completely restrictive) and 100 (completely permissive) for the gun laws of all 50 states taken from a reference guide for gun owners traveling between states from 1998 to 2015. Gun ownership was estimated annually as the percentage of suicides committed with firearms in each state. Main outcome measure Mass shootings were defined as independent events in which four or more people were killed by a firearm. Data from the Federal Bureau of Investigation’s Uniform Crime Reporting System from 1998–2015 were used to calculate annual rates of mass shootings in each state. Mass shooting events and rates were further separated into those where the victims were immediate family members or partners (domestic) and those where the victims had other relationships with the perpetrator (non-domestic). Results Fully adjusted regression analyses showed that a 10 unit increase in state gun law permissiveness was associated with a significant 11.5% (95% confidence interval 4.2% to 19.3%, P=0.002) higher rate of mass shootings. A 10% increase in state gun ownership was associated with a significant 35.1% (12.7% to 62.7%, P=0.001) higher rate of mass shootings. Partially adjusted regression analyses produced similar results, as did analyses restricted to domestic and non-domestic mass shootings. Conclusions States with more permissive gun laws and greater gun ownership had higher rates of mass shootings, and a growing divide appears to be emerging between restrictive and permissive states.

The methodology deserves critique.

Gun Permissiveness

We used the 1998–2015 edition of the Traveler’s Guide to the Firearms Laws of the Fifty States to obtain the independent variable of interest, an annual restrictiveness-permissiveness scale of US gun laws for each state

This sounds like a fascinating book for gun owners to have on a shelf. You can apparently buy it here, and it’s updated yearly. Here I must state I’m not being compensated in any way for driving them traffic. (this a Medium rule, and a good one) I have not read this book, but from what I gather from the BMJ report, they basically take all the following factors, weight them, and issue a composite score:

The report considers more than 13 factors in developing the score, including: standard firearms ownership and permit requirements; if semi-automatic, high capacity magazines, machine guns, and suppressors are permitted or restricted; if the firearms laws across the state vary widely; if the state employs a right to self-defense, ability to conceal, ability to open and vehicle carry, ability to conceal carry in state parks, or whether a gun permittee can carry in a restaurant serving alcohol; whether there is a duty to notify law enforcement of permit status; and if one can keep a gun in their vehicle at colleges and K-12 schools (primary and secondary schools).

The first thing we need to point out here, is that the Traveler’s Guide isn’t ranking states by how hard it is to become a gun owner. They are ranking states by how big of a pain in the ass the government makes gun ownership. We’ll come back to this.

Composite scores are always suspect when you get into data analyses, because they depend on arbitrary decisions by the author about how to weight stuff. But we won’t lay too much criticism on the BMJ authors here, because this was probably the best source available for what they were trying to do. I would be more alarmed if they chose a politically biased source, such as the rankings from The Gifford Center or Guns and Ammo.

Gun Ownership Rate by Suicide Proxy

Gun ownership is not directly surveyed across all 50 states each year in the US. A review of over 24 gun ownership indicators found that the percentage of suicides committed with a firearm was the best measure for estimating gun ownership by state.17 This has also been verified in several other studies across different regions,1819202122 in which the percentage of suicides committed with a firearm was shown to be highly correlated with the proportion of households reporting gun ownership (across 21 US states r=0.90,23 across nine census regions r=0.9324).

Gun researchers almost always use suicide proxy to determine ownership rate, which works basically like this. You plot the ratio of gun suicides to total suicides, compare it against accurate polls of gun ownership, develop a best fit relationship, and then use that best fit equation to back-figure gun ownership. I don’t like it, for several reasons.

The main problem in using suicide proxy is confounders. Overall suicide rates vary by state, and not only with gun ownership, but with many cultural factors, which flow from socioeconomics and mental health. At its root, the use of the suicide proxy to approximate gun ownership, especially without any other controls, rides on the presumption that the mental health profiles of each state are the same. This may not be a bad presumption for many generalized studies of gun violence, particularly at the national level. But if you’re going to specifically study mass shootings, you’ve got a big problem. Mass shootings are deeply linked to mental health, so any metric that’s also linked to mental health is going to have hidden effects on your data that you need to control against. Your results will be kinked.

There’s been a lot of study about how to un-kink this, and Dr. Michael Siegal’s name seems to crop up a lot in the journals discussing the topic. The study in our second article on the suicide problem, authored in part by Siegal, uses the rate of issuance of hunting licenses to unkink it. Here’s another study that used subscriptions to Guns and Ammo magazine to unkink it. Different researchers use different methods.

Suicide proxy is unfortunately about the best thing statisticians have for approximating gun ownership rate, so the researchers were basically stuck with it here, but they don’t appear to have done anything to untangle the mental health confounder. And the mental health confounder is very important for mass shootings.

Mass Shooting Definition

Mass shootings were defined as independent events in which four or more people were killed by a firearm.

This is a reasonable definition, and a thing worth of research as-defined, but it is not at all what the public thinks of when they hear the words “mass shooting.” These incidents are probably dominated by “drug deals gone wrong,” and mostly not “suburban soccer moms and kids getting gunned down randomly.”

Further, mass shootings are a tiny sliver of the gun death problem. They’re almost literally not worth mentioning. From a prior HWFO article:

This entire study is about the tiny yellow sliver in the smallest pie chart on the bottom left. Remember this, we will return to it.

Multivariate Assessment

We included the following annual measures of state-level characteristics in our analyses: median household income, percent high school graduation, percent female headed households, percent in poverty, percent unemployment, incarceration rate, and percent white.

The study was multivariate, which is good. The only way to garner anything meaningful in this research space is to separate out the different factors that impact gun violence. It would be nice if violence researchers spent more time on the factors that are more determinate, such as wealth inequality, but those don’t fit the media narratives particularly well. It’s a bit curious that they chose “percent white” instead of “percent black” in their list, and I wonder why. Asians and Latinos have tremendously lower homicide victimization rates than African Americans, so that makes me wonder if the choice wasn’t driven by political correctness instead of good science. Black population ratio is five times more predictive of overall homicide rate than gun ownership rate.

But the real howler, the worst thing the study did, was here:

Because restrictiveness-permissiveness of state gun laws and state gun ownership were highly and significantly correlated (Pearson’s r 0.79, P<0.001) and interdependent, we did not include them in the same regression models.

What?

I’ll say this again, in italics this time:

What?

The entire point of a multivariate analysis is to control significantly correlated confounders. And there’s a huge link between gun ownership and restrictive gun laws. It’s also not “interdependent,” the dependence is almost entirely one-way. It’s called the ballot box.

As we mentioned above, the Traveler’s Guide isn’t rating how difficult it is to become a gun owner, it’s rating what a pain in the rear the government makes gun ownership. And gun owners are, in isolation, always going to vote against the government making their lives a bigger pain in the rear. Increased regulations don’t reduce gun ownership, gun ownership tempers and controls regulations. The correlation is strong. There is absolutely no reason to not include gun ownership as a confounder in their analysis, particularly given that the researchers acknowledge the strength of the correlation.

Let’s step sideways for a second and look at a multivariate analysis from a prior HWFO article, penned in part by the aforementioned Dr. Siegal:

From Siegal, Ross, and King, in AJPH, here

The three biggest potential confounders for the BMI study, as identified by Siegal et al. in the above table, are black population ratio, Gini Coefficient (a measure of wealth inequality), and gun ownership rate. Additional minor confounders were violent crime rate, nonviolent crime rate, and incarceration rate. The BMJ study controlled for “percentage black” in a probably poor way, controlled for Gini Coefficient by looking at poverty rate and unemployment rate, and controlled for incarceration rate. But they chose to ignore gun ownership in their multivariate analysis, and instead did a separate analysis on it.

Why would they do this? Maybe we can get a clue from their results: (emphasis mine)

Results Fully adjusted regression analyses showed that a 10 unit increase in state gun law permissiveness was associated with a significant 11.5% (95% confidence interval 4.2% to 19.3%, P=0.002) higher rate of mass shootings. A 10% increase in state gun ownership was associated with a significant 35.1% (12.7% to 62.7%, P=0.001) higher rate of mass shootings. Partially adjusted regression analyses produced similar results, as did analyses restricted to domestic and non-domestic mass shootings.

That certainly seems like gun ownership rate is far more predictive than permissiveness, and we already know ownership rate and permissiveness are linked via the ballot box. But apples are not oranges, and these comparisons don’t marry up unless we can normalize them, so they use the same incremental unit in comparison.

Presuming we accept the suicide proxy methodology, the national gun ownership rate in the USA varies from 20% to 70% — a variation of 50% across the data set. The “Traveler’s Guide to the Firearms Laws of the Fifty States” rating system assigns gun permissiveness values to the states ranging from 20 to 100, a variation of 80 “units.”

A variation of 1/8th across the permissiveness range changed the mass shooting incidence rate by 11.5%. A variation of 1/5th across the ownership range changed the mass shooting incidence rate by 35.1%. If you normalize these figures by range increment, you see that (gun ownership rate) is 1.9 times more predictive than (gun law permissiveness) to mass shooter rate.

That’s a noticeable difference. Gun ownership is twice as predictive. This leads me to highly suspect that if they’d done their job properly, and included ownership rate itself as a confounder in their multivariate analysis to compensate for the voting booth effect, the correlation between (gun law permissiveness) and (mass shooter rate) may be negative, and the positive correlation they do show is purely an artifact of higher ownership rates.

I wonder whether they actually did this calculation originally, didn’t like the answer, and then went back and stripped the gun ownership confounder out of the multivariate analysis to get graphs that would play better on Vox and Yahoo. I have no way to know.

Without controlling for ownership properly, we don’t know anything about whether gun laws have any effect at all. It seems they may be negatively correlated. How could that be? Perhaps, again, the ballot box. Perhaps a state with a higher mass shooting rate than it would ordinarily expect given their gun ownership rate is more inclined to pass restrictive laws.

It’s just an idea.

The Rates are Fabulously Low Anyway

Let’s look at their graphs.

You can plainly see the effect we spoke about in the prior section. The gun ownership line is steeper than the gun permissiveness line, and ownership vs permissiveness are linearly correlated. It shouldn’t be hard to control permissiveness against ownership. Anyone worth their salt could do it in about ten minutes with their data. But let’s look closer at the axis themselves.

“Mass shootings per million people”

Million.

That rate is fabulously low. The range peaks at 0.3, with a median (eyeballing) of around 0.1. The states in their data set have a mass shooting rate of around one incident per ten million people, and that’s even with the rather broad definition they chose (four or more people shot) instead of the colloquial media definition (unhinged assholes randomly shooting kids and soccer moms).

Put this number in perspective. Between 40 and 50 people per year die from lightning strikes in the United States, for a national death-by-lightning rate of around 1.38 per ten million people. In the average US town, a mass shooter incident as defined is less likely that someone not only getting hit by lightning, but dying from the lightning strike. And only one in 12 lightning strikes is deadly.

But we don’t realize how rare these things are, as body politic, because we’re glued to our phones addicted to outrage porn. This study is just feeding the outrage engine, and seems quite honestly to have committed at least one egregious error in its multivariate analysis while doing so. Maybe more.

Conclusions

From the abstract:

Conclusions States with more permissive gun laws and greater gun ownership had higher rates of mass shootings, and a growing divide appears to be emerging between restrictive and permissive states.

This is absolutely technically correct, given their methodology. But a more proper and honest conclusion statement would probably have been this:

The fantastically rare instance of four or more people dying in a gun homicide, which makes up such a small fraction of gun deaths that it’s quite honestly barely worth mentioning, is correlated to both gun ownership and gun law permissiveness. The second correlation to gun law permissiveness is very likely just an artifact of the first correlation, because gun owners vote in their best interests. Gun laws likely have no correlation (possibly a negative correlation) to the rate of incidences of mass shootings.

Nothing in this study says more gun laws would reduce mass shootings. If they issue a revision of their multivariate analysis which controls against gun ownership rate itself, it might, but it’s very unlikely based on the data they presented.

Which may be why they didn’t do it.

I’d love to read an updated study though.