I was gonna give this post the title, Stat Rage More Severe in the Presence of First-Class Journals, but then I thought I’d keep it simple.

Chapter 1. Background

OK, here’s what happened. A couple weeks ago someone pointed me to a low-quality paper that appeared in PPNAS (the prestigious Proceedings of the National Academy of Sciences), edited by the same person who approved the notorious himmicanes paper and who had earlier published a paper with one of the authors of the notoriously power pose paper. This new article was an attempt to understand the sources of “air rage” but unfortunately the actual analysis was a big uninterpretable multiple regression on some observational data.

Just to get the criticisms out of the way, here’s what I wrote, explaining why I couldn’t believe any of the claims in that article:

The interpretation of zillions of regression coefficients, each one controlling for all the others. For example, “As predicted, front boarding of planes predicted 2.18-times greater odds of an economy cabin incident than middle boarding (P = 0.005; model 2), an effect equivalent to an additional 5-h and 58-min flight delay (0.7772 front boarding/0.1305 delay hours).” What does it all mean? Who cares! Story time: “We argue that exposure to both physical and situational inequality can result in antisocial behavior. . . . even temporary exposure to physical inequality—being literally placed in one’s “class” (economy class) for the duration of a flight—relates to antisocial behavior . . .” A charming reference in the abstract to testing of predictions, even though no predictions were supplied before the data were analyzed. They report a rate of incidents of 1.58 per thousand flights in economy seats on flights with first class, .14 per thousand flights in economy seats with no first class, and .31 per thousand flights in first class. It seems like these numbers are per flight, not per passenger, but that can’t be right: lots more people are in economy class than in first class, and flights with first class seats tend to be in bigger planes than flights with no first class seats. This isn’t as bad as the himmicanes analysis but it displays a similar incoherence. I didn’t explain all these points in detail—this was a blog post, not a textbook or even a referee report—but it was all there. To spell it out: My first quoted paragraph above addressed the problem of the regression coefficients, which is the same problem you noted in your comment. My second quoted paragraph is relevant in that the paper makes claims about human behavior which are not supported by their data. My third quoted paragraph pointed out the non-preregistered nature of the analysis. As always, preregistration is not required, but when this sort of completely open-ended study is not preregistered, this calls into question all claims of prediction accuracy and p-values. My fourth and fifth paragraphs address the point that the direct comparisons presented in the paper are uninterpretable. Finally, the data are unavailable so it is impossible for an outsider to evaluate any of these claims. If the data were public, I’d recommend publication under a much lower standard, because once the data are out there, others could do their own analyses.

For another take on data problems with the “air rage” paper, see this post by John Walton.

Chapter 2. First mention of NPR

When posting on this study, I threw gratuitous shade at one of America’s most trusted news sources in my “tl;dr summary”:

NPR will love this paper. It directly targets their demographic of people who are rich enough to fly a lot but not rich enough to fly first class, and who think that inequality is the cause of the world’s ills.

The next day I posted a roundup of media outlets that’d fallen for this story, including CNN, the LA Times, and ABC News, along with respected tech sources Science and BoingBoing. I discussed the selection bias that occurs when the best science reporters realize this study is empty and don’t report it, while everyday journalists just follow the PPNAS label and don’t even think there could be a problem. All jokes about “stat rage” aside, this is a big problem in that consumers of the news only see the sucker takes, never the knowledge.

But NPR wasn’t included in that media roundup, hence I wrote “I was unfair to NPR,” and commenter Sepp asked,

Why the dig at NPR? And why the implication that NPR listeners cannot distinguish good scientific articles from bad ones that agree with listeners’ values? On that note, why the implicit indictment of said values (i.e. the desire to reduce inequality, etc.)? I find these statements saddening and confusing.

Chapter 3. The return of NPR

But then, after all that, NPR bit on the story—multiple times!

I’m sad to say that our public radio network lived up to its reputation. And I really am sad. I’d be much happier to report that they showed admirable skepticism and restraint. But they didn’t:

– Wait Wait Don’t Tell Me (or so I’ve been told; I haven’t seen the transcript so maybe they were actually mocking the study; I can only hope.)

– Planet Money:

– And finally, this from Alva Noë:

The nation’s finest news source, indeed. It could’ve been worse: it could’ve been mentioned on NPR’s evening news show, but still, it’s disappointing.

Chapter 4. Summary

I return to my original statement:

NPR will love this paper. It directly targets their demographic of people who are rich enough to fly a lot but not rich enough to fly first class, and who think that inequality is the cause of the world’s ills.

NPR was not the only prestige outlet to be fooled.

Science magazine fell for this story (“Air rage? Blame the first-class cabin”). No surprise they were duped, I guess: tabloids gotta stick together.

But I was disappointed to see the usually-skeptical Economist take the bait too (“Resentment of first-class passengers can be a cause of air rage”). The Economist is no great enemy of inequality so they had no particular political reason to like this one. I guess they got conned by the PPNAS label.

Good job, PPNAS: another short-term win for the PPNAS publicity machine, another long-term loss for your reputation. Or, should I say, medium-term, as I sill have hope that you will clean up your act someday.

Let me conclude with this, from a commenter on the Economist article who understands this better than the Economist’s own reporter:

I think NPR and these other news outlets can do better. And really, who goes into journalism to reheat press releases, anyway?

And, hey, to the authors of the paper: It’s ok. Everyone makes mistakes. Statistics is hard. I’m sure you were intending to do good science here. I’m not quite sure what to recommend for you. For this particular line of study: Sure, I’d recommend you cut your losses and release a short statement recognizing that the data don’t support your claims. After that? Maybe bring in a couple collaborators, one who knows about airlines and one who knows about causal inference. Sure, that takes work, but it’s kind of a necessity if (a) you want to learn anything useful from these data, and (b) you’re not willing or able to just make the data public. For future research on other topics: Hmmm, I guess my quick suggestion for any future paper is to present it at seminars in a few econ departments. Economists are pretty tough. They don’t catch every error but they might’ve caught these. And, for God’s sake, don’t submit to PPNAS anymore. Look what happened last time. In all sincerity, I wish you the best, and I urge you to reject the lure of the quick publication. PPNAS isn’t doing you any favors by publishing work like this (except, I guess, indirectly, in that now you’re getting some free advice from me). It’s a dysfunctional relationship we have here, between journals that seek publicity, news organizations that are all too willing to essentially run press releases, and researchers who often just don’t know better, and are led to believe that anything with “p less than .05” that’s published in a journal is good for them. Time to jump off the merry-go-round.

P.S. I wrote this post several months ago—this blog’s on a lag—and it just happened to appear today.