Yesterday we discussed a sad but all-too-familiar story of a little research project that got published and hyped beyond recognition.

The published paper was called, “The more you play, the more aggressive you become: A long-term experimental study of cumulative violent video game effects on hostile expectations and aggressive behavior,” but actually that title was false: There was no long-term study. As reported in the article, the study lasted three days, and in each day the measurements of outcomes may well have been conducted immediately after the experimental treatment. The exact details were not made clear in the article, but there’s no way this was a long-term study.

Yesterday I discussed how mistake could’ve made it through peer-review. Today I want to talk more generally about incentives.

But first, let’s step back a moment . . .

Before discussing any further, let’s just consider how ridiculous this all is. A paper was published by legitimate researchers (one of them is the Margaret Hall and Robert Randal Rinehart Chair of Mass Communication at Ohio State University) in a legitimate journal (Journal of Experimental Social Psychology, impact factor 2.2), it receives over 100 citations and national press coverage—and the title is flat-out wrong. (3 days is not “long term.”)

And, the most amazing thing of all . . . nobody noticed! An experimental science paper that mischaracterizes its experiment in the title—that’s roughly equivalent to a math paper declaring 2+2=5.

You know that saying, “The scandal isn’t what’s illegal, the scandal is what’s legal”?

Something similar here. The scandal is not that somewhere, in some journal, some authors screwed up and mis-titled their paper and the reviewers didn’t notice. Mistakes happen. I’ve messed up in published work in lots of different ways. No, the scandal is that this huge error was sitting there, in plain view, for five years! And nobody noticed. Or, I should say, if anybody noticed, I never heard about it. I guess that’s part of the problem right there, that it’s not so easy to correct the published record.

Incentives, incentives

My second-favorite bit of the above-linked article:

Another limitation is that our experiment lasted only three days. We wish we could have conducted a longer experimental study, but that was not possible for practical and ethical reasons.

Fine. If you can only do a 3-day study, just change “long-term” to “3 day” or, maybe, “5 minute” in the title of your paper. How hard is it, really, to just say what you really did? I guess, as commenters keep saying, it’s the incentives. Label your paper as a 3-day study and you might not get the citations and influence that you’ll get by calling it “long term.”

From the webpage of one of the authors of this paper:

So, consider three alternatives in designing and writing up this study:

1. Do a truly long-term study, following a group of people for a few years. Hmmm, that’s lots of work, don’t wanna do that!

2. Do a 3-day study, each day redoing the intervention and testing immediately after. This is inexpensive and likely to get solid results—but it’s not very interesting, will be hard to get published in a good journal and hard to get publicity later on.

3. Do a 3-day study, each day redoing the intervention and testing immediately after. But then put “long-term” in the title and hope for the best! This gives you all the convenience of the easy option, but with the potential for the major citations and media exposure that would be appropriate for an actual long-term study.

The incentives favor option 3.

You don’t have to be a “bad guy” . . .

Apparently these are the rules of the game, at least in some areas of science: You do an experiment which somewhere gives you statistical significance, you get the paper accepted at a journal, and then you misrepresent what you’ve learned, in the title of the paper and in the publicity material. (See the end of this post for an example where, in a single sentence of the publicity materials, one of the authors made 3 different claims, none of which are supported by the data in the research article.)

This sort of behavior is, in my opinion, destructive to science. But it happens all the time, to the extent that I doubt the authors even realized what they are doing. After all, they may well be personally convinced that their research hypotheses are true, thus, in their view, they may be saying true statements.

The idea that it may be true but it’s not supported by the data—the distinction between truth and evidence—that seems to be difficult for a lot of people.

I strongly doubt these researchers are trying to misrepresent their evidence. Indeed, once you become aware of the misrepresentation, and once you become aware of the distinction between truth and evidence, it becomes difficult to grossly misrepresent the evidence, if for no other reason than that it seems so obvious and embarrassing.

So, it’s not that these researchers really think that 3 days, or 5 minutes, is “long-term.” They just feel they’ve made a general discovery, they have no reason not to believe they’ve identified a long-term effect, and they’re reporting the truth, as they see it.

The trouble is that lots of outsiders—journalists and the general public, policymakers, and other scientists—might naively take these unsupported claims at face value, and think that Hasan et al. really did conduct “a long-term experimental study of cumulative violent video game effects on hostile expectations and aggressive behavior.” Which they didn’t.

There are so many incentives for researchers to misrepresent their data, and at the same time we have to deal with ostriches who say things like, “The replication rate in psychology is quite high—indeed, it is statistically indistinguishable from 100%.” So, yeah, there’s a reason that we keep screaming about all this.

I don’t care so much about this particular paper, which I’d never even heard of until someone sent me an anonymous email about it. But I do care about the larger issue, which is what’s happened to the scientific literature, when it’s considered OK, and unremarkable, to misrepresent your study in the title of your paper.

Like a harbor clotted with sunken vessels.