The US DOES Have Fewer Annual Deaths From Mass Shootings than Some European Countries, and Whether or not Snopes Likes the Numbers Doesn’t Change That Johnny Steen Follow Jul 23, 2018 · 11 min read

An older article is making the rounds on social media again; on the one hand in response to the fact that it’s gun week again, and on the other hand because Snopes finally got around to spinning it in their direction. Snopes starts by making a few good points, but quickly veer off into the worst display of “statisticking” I’ve ever seen, and all so that they can justify their patently absurd conclusion that every country except the US has zero gun deaths.

I’ve noticed that, when an outlet reports a statistic, they tend to choose the simplest statistic that proves the point they set out to make. That’s why, when CNN says that “hourly” wages for non-supervisory workers haven’t gone up, it’s because removing any of those qualifications results in statistics that don’t prove their “the sky is falling” narrative, which is how they sell clicks.

Reporters aren’t scientists. They don’t use the scientific method; rather, they say “I’d like to write about X”, and then set out to find supporting evidence. This is fine, really, because for every reporter trying to prove X, there’s another one contradicting that. The more correct argument should win over the reasonable populace. Or, more often, two stories that seem contradictory aren’t, and a reasonable reader with access to both walks away with a far more nuanced view of a complicated issue. This is the adversarial method of fact finding and is also how court works.

But as far as simple goes, there’s nothing much simpler than taking an average, and for reasons we’ll explore here, Snopes has a problem with that. Their argument can be summed up with the following quote:

Where the United States saw at least twelve mass shooting deaths every year between 2009 and 2015, some of the other countries on Lott’s list experienced one or two rare but very high-casualty shootings.

What they fail to address is how much smaller all of these countries are (by population) than the United States. It should not take much convincing that a larger country will have more shootings simply because it has more people; imagine, for instance, you made a perfect clone of the United States and dropped it on some other planet on some other solar system. You’ve doubled the people, and you’ve doubled the number of shootings. The number of shootings didn’t double because of bad public policy, because the policy didn’t change; only the number of people.

But rather than do the simple thing, which is report, for example, a) the number of shootings per year per million people and b) the average number of deaths per shooting (both of which are invariant to population and should depend only on culture and policy), they do the ridiculously complex task of:

For each country, add up the number of deaths by mass shooting in each year. Then, for each country, take these seven numbers and report their median.

Do this, and surprise, surprise! You find every country other than the US has no mass shootings! (Spoiler: but you also find the US has no mass shootings either, if you either: a) replace “years” by “weeks” in this algorithm, or b) apply the procedure on a state-by-state basis instead of the US as a whole.)

To their credit, they at least attempt to justify their alternative statistics, which you have to do if you’re being intellectually honest. If you believe in your method, you have to allow someone like me to tear it apart.

The risk is that it’s complete garbage and someone like me will tear it apart, because this needlessly complex procedure was designed to imply that every country other than the US has no mass shootings. That’s patently false.

They could’ve argued the data was made up, if they thought that was the case. They didn’t. They accepted the data as accurate. Instead, they questioned the methods of interpreting the data — and that’s where they got it wrong.

I say this a lot: the job of a fact checker is to verify the facts. The raw data. It is not the job of a fact checker to make sure everyone on the internet does the right math problem to come up with the correct opinion.

What they got right

The average number of deaths per capita per year due to mass shootings doesn’t tell the entire story, sure. It doesn’t differentiate between rare, deadly shootings and frequent, less deadly ones.

I’m not sure how that distinction should affect public policy, but people do seem to prefer consolidating their grief into one big incident. So there’s that. Having mass shootings be more rare does indicate that fewer people wish to commit a mass shooting and also have access to a gun. But having each mass shooting be more deadly indicates that the victims were more poorly defended. Whichever outcome you think is better is a matter of personal preference, because math does not provide a way of comparing two pairs of numbers.

As much as it may bother you to have a personal preference when it comes to mass shootings, when the choice is “which country is better” then that’s the decision you have to make. You do not get to use fancy (wrong) statistics to make one of them zero to make yourself feel better for having a preference.

“Bin by Year and Take the Median” is Wrong

The argument is made that Norway, ranking at the top in number of deaths per capita per year, was given an unfair shake because all of those deaths happened in one shooting.

They have a point.

But then they go and use an even worse statistic to make a conclusion so ridiculously in their own favor that I haven’t peeled my palm off of my face since I read it. The remedy for Snopes’ problem is to simply say how rare mass shootings are. Simple. They didn’t do that. Instead, they claimed you should look at the median of the numbers of deaths by mass shootings as aggregated by year.

For starters, the median is a bit arbitrary; in this context a median of zero only indicates that there are mass shootings less than 50% of the years observed. Why 50%? Certainly 50% is an important threshold in many contexts, but there’s nothing special about it in every context (unless you’d like to imagine a world where shootings happen that often) and the mere fact that we gave 50% the name “median” doesn’t make it less arbitrary. It has its use, of course; this isn’t one of them.

Moreover, why take the median over the numbers of deaths per year? If we bin shootings by number of deaths per day instead of per year, America’s median is also zero. What’s special about years over days, besides that there are more of one than the other? Why not 100 day windows, or 500 days? If our choice of time frame has such a profound effect on the outcome, one must be precise about why one time frame is chosen over another. (The answer is, you don’t. You bin the number of deaths per shooting. But more on that later.)

That’s the rub with using their analysis: it doesn’t scale with population. That’s not an issue as long as what you’re observing is so frequent that you have a lot of data. But the situation here (thank God) is that we have maybe one or two events over a seven year period. The authors of the Snopes article chose the median specifically because this is the case — they claim that what is more comparable to a rounding error than a statistic is more fair because they like the results it gives, but there is no statistical theory that justifies that kind of reasoning.

Besides the time window, there’s a spatial window as well. Each European country is roughly on the same scale as the US’s larger states, and the US as a whole is roughly on scale with the European countries in aggregate. As the CPRC’s response points out, if we look at the mass shooting deaths by state and apply the same “bin by year and take the median” procedure that Snopes taught us, we find that every state’s median yearly other than California is also zero.

Applying Simple Models to Complex Phenomena

Snopes’ justification mostly boils down to the word “consistency”. Because one number doesn’t accurately reflect how rare mass shootings are in each European country.

They call Norway, for example, an “outlier”, which is just another example of why I’d love to see journalists be required to get a license to put numbers in their articles. I’m glad the one thing they took away from Stats 099 For Journalism Majors is to throw out data, but there are only a few scenarios in which you should do that, and in every scenario there is a 200-line proof and a graduate level course’s worth of justification for doing so, and not a single one of them is “I have to in order to prove my point”.

We discount outliers because we don’t want anomalies to affect our analysis of ordinary behavior. But thankfully, mass shootings, like terrorist attacks, are the anomalies. All of them are “outliers”. We do not throw them out.

The statistics we use must be justified by the model we’re assuming generated the data. (For instance, the “trimmed mean” estimator, wherein you throw out the top and bottom few data points, can be proven to have smaller variance if the data are normally distributed.) In this case, the model has to account for two different things:

The rate at which events happen The severity of any particular event

Statisticians have a name for models that describe this kind of behavior: it is a compound Poisson process. What Snopes termed “consistency” would better be called “continuity”, and compound Poisson processes are discontinuous regardless of how frequently events happen.

There are entire books written on compound Poisson processes (CPP), but to the point of this article, I’ll just say one thing: there is no statistical theory that says you should aggregate all of the events in a particular time window, and then aggregate them again across all time windows. But that’s what Snopes has done; they:

looked at each year as a time window, added up all the deaths in each year, and then took the median of that.

That kind of statistic does not provide meaningful information about any of a CPP’s parameters. It makes no statistical sense, and there is no principled reason to do it.

The actual solution

This may scare those journalists who want a single nugget of fact to put in their headlines, but there’s really only one way to be fair to the type of model mass shootings follow: you have to report two numbers per country. I know that almost no reporters ever do this, but compound Poisson processes have two parameters that each have a profound effect on quality of life in this context, so reporting one number isn’t telling the whole story.

This was Snopes’s entire problem to begin with, so good on them for that. But still bad on them for their proposed solution.

The statistics for each country should address the following separately:

Average number of shootings happening per year per million population. (Explained below.) The average number of deaths per shooting.

Why separately? Because they’re separate things. That’s all. If you still want the total number of deaths per year, multiply the two.

If you love the median so much, you could look at the median number of deaths per shooting, instead. But not the median total number of deaths by shooting per year. In the case of Norway, that brings their median up to 69 because they only had one. (The average remains unchanged.) Somehow I doubt Snopes would still have a strong preference for the median when they can’t end their article with the pseudostatistical “f — k you” that they did in this case.

But at least Snopes can take relief in the fact that they can pair that number “69” with the other relevant parameter, which is the rate at which shootings happen.

Which, lo and behold, the CPRC provides! Although I think they could have been clearer, they report this figure as 0.197 per million people per seven years. And that doesn’t really prove Snopes’s point either (the US is twelfth in that aspect), so they don’t use it. They might have argued that this number shouldn’t be adjusted for population; even if they didn’t, I’ve heard the argument. So I will address it next…

A Note About Per Capita Statistics

Some people were confused by the Snopes article and thought that Snopes was calling the practice of reporting “per million” was wrong. Not only is that ridiculous, but to Snopes’ credit, that’s not what they were saying, either.

One should always ask oneself if a particular number could be naturally expected to scale with population. Things like the number of deaths due to heart attacks certainly should. If so, that number should be reported per capita so that the conclusions aren’t biased by the fact that one country simply has more people than another.

Given that, all else equal, every person probably has roughly the same likelihood of committing a mass shooting, I claim that the rate of mass shootings should be reported per capita (or per million). I also claim the severity of each shooting should not be reported per capita; the amount of damage a shooter can do doesn’t really have anything to do with how big the country is. (Population density, maybe, but then the issue of combining these gets complicated.)

Since “rate” already implies “per unit time” (like per year), that means the proper number would be reported as something like, “X shootings per year per million people”. That’s a lot of “per”s, but if you don’t do this, then X is just a measure of how big each country is. This way, for example, each US state’s particular X is roughly the same, and the US’s X is the average of all the states’ Xs. Recall that a major problem with Snopes’s method was that every state had a “Snopestistic” of 0 if considered separately. That anomaly doesn’t happen now.

I do have to call out the CPRC on their lack of clarity in presenting the “frequency” of mass shootings. Based on the data, they must have just taken the number of shootings over the entire period and divided by the population (in millions). They should have also divided by the length of time the data were collected over, which in this case is seven years. Since data for every country were collected over the same window of time, the comparison is still valid; however, a number “per seven years” is weird. If they’d put units in their column headers like good scientists, they probably would have noticed that too.

Snopes Should Feel Bad

Snopes didn’t go the reasonable route, they went the “the conclusion is the exact opposite and anyone who disagrees is wrong” route. They went on pretending that a complex issue with an equally complex statistical distribution can be easily compared, and their way of comparing them is the only way. That’s exactly what they said was wrong with CPRC’s analysis.

Sure, a lot of people look at just the CPRC and conclude that mass shootings are worse in Europe than in the US. I don’t think there’s enough information to conclude that.

But Snopes could have taken the reasonable route.

They didn’t. They went full swing in the opposite direction and insisted that mass shootings are “a uniquely American problem”.

Because nobody can have a reasonable opinion anymore. Everybody’s gotta say the sky is falling, and that this group of people over here is causing it. You’re either OK with an innocent person being shot by the police, or you’re a cop-hating recidivist miscreant who doesn’t appreciate the sacrifices law enforcement officers make for your family. You either believe gun control has turned all of Europe into a utopia where nobody kills anybody or you’re an NRA funded shill. Nobody’s allowed to argue with the extremist elements of any political party without getting mobbed.

Just stop already. Math isn’t a child for the divorced political tribes of society to fight over the custody of, and I won’t stand for it anymore.