David Palmer writes:

If you need yet another study to look at, check this out: “Reduction in Firearm Injuries during NRA Annual Conventions.” <<1% of gun owners attend a convention, and gun injuries drop 20%.

Sounds fishy to me. It was published in the prestigious, practically-impossible-to-publish-in New England Journal of Medicine . . . but (a) these super-selective medical journals are so selective, that whatever they do publish, is a bit of a crapshoot, and (b) medical and public health journals seem to have a soft spot for research that has a (U.S.-style) liberal story (for example here and here) or (c) maybe just anything that can grab a headline (for example here).

Here’s the summary data from this gun injury paper:

And here’s the adjusted version:

The standard errors in the two plots are different but the point estimates appear to be identical. I assume there was some mistake in the preparation of one or both of these graphs. In any case, the standard errors hardly matter as you can see the variation from point to point. I’d just like to see each year and each day rather than these aggregates.

The real problem, though, is what was identified by Palmer, above: the idea that a sequestering of less than 1% of gun owners would lead to a 20% drop in gun injuries. Much more direct to observe that the rate of gun injuries varies a lot from day to day, and from week to week, and from year to year, for all sorts of reasons—and those sources of variation don’t happen to cancel out, over the 9-year period of this study.

What I’d really like to see are the data for all 365 or 366 days of every year, and then we could do this sort of analysis. In the case of the gun injuries data, I don’t know how hard it would be to get these data. Once they went to the trouble of getting the numbers used in the article, do the data for all the other days of the year just come for free? Or would you need to do an equal amount of effort for every new day in the dataset? I guess it would help to get a sense of how much it would cost to put together the data for every day of the year.

In the meantime it would be helpful for the authors to share all their raw data. I know this isn’t standard practice—I don’t share the raw data for most of my papers, not because I’m trying to hide anything but because it would just seem like too much trouble. We should just be all moving toward a norm of full data sharing. In this particular case, the key data are something like 189 numbers (9 years x 7 weeks x 3 days during the week); that would be a start.

The journal article (and also news reports) also picks up some interactions, but given that the main effects are iffy, I don’t think we need to even really think about the interactions at all. I don’t believe any of the standard errors either (there’s some super-complicated regression procedure with many different knobs to be set, but really there are only 7 independent data points per year, as can be seen in the above graph), for that matter.

A quick search turned up this news article from Scientific American which said, “The design of this study only identifies associations, not precise cause-and-effect relationships, and so is unable to ascertain that the observed injury drop on convention days came about because NRA members are not using their weapons. But several study details support this explanation. . . .” So I think I should clarify something. Yes, correlation does not imply causation in this sort of observational study. But, beyond that, the correlations themselves are not so clear, and the effect sizes just don’t line up.

You can look at this another way. Suppose that you might expect the NRA convention to actually be causing a 1% decline in gun injuries. Then you can work through a design analysis and find that you’re in the notorious “power = .06” situation, i.e. the kangaroo problem. Your study is dead on arrival.

That said, I could be wrong. The research article does give some reasons why such a huge effect could be possible. And there’s nothing wrong with opening up some data and publishing what you’ve found. The analysis has a bunch of iffy steps, but that’s ok too, as long as you share your raw data so other people can do their own analyses. In this day and age, there’s no real excuse for not sharing the data. Again, I haven’t been always so good at making my own raw data accessible, so I say all this not as a criticism of the authors of the above paper but rather as a suggested step forward.

To return to a common theme: Publication of interesting data is a great thing to do. The mistake is to take a data pattern too seriously, just because:

(1) Some researcher somewhere did some analysis resulting in “p less and 0.05,” and

(2) Some legitimate journal somewhere happened to publish this result.

We know now that (1) there are lots and lots of ways to get “p less than 0.05” from noise alone, or, more precisely, from patterns that are so highly variable and context dependent to be essentially uninterpretable given available data, and (2) journals publish all sorts of iffy claims all the time.

So let’s try to move on, already. Please. My point is not to slam the authors of the above article, who put in some hard work in data collection and analysis and were surely doing their best. My problem is with the system of scientific publication and publicity, in which these sorts of speculative analyses are uncritically hyped