E. J. Wagenmakers points me to a delightful bit of silliness from PPNAS, “Hunger promotes acquisition of nonfood objects,” by Alison Jing Xu, Norbert Schwarz, and Robert Wyer. It has everything we’re used to seeing in this literature: small-N, between-subject designs, comparisons of significant to non-significant, and enough researcher degrees of freedom to buy Uri Simosohn a lighthouse on the Uruguayan Riviera.

But this was my favorite part:

Participants in study 2 (n = 77) were recruited during lunch time (between 11:30 AM and 2:00 PM) either when they were entering a campus café or when they had eaten and were about to leave. . . . Participants reported being hungrier when they walked into the café (mean = 7.38, SD = 2.20) than when they walked out [mean = 1.53, SD = 2.70, F(1, 75) = 107.68, P < 0.001].

Ya think?

But seriously, folks . . .

To me, the most interesting thing about this paper is that it’s so routine, nothing special at all. Published in PPNAS? Check. Edited by bigshot psychology professor (in this case, Richard Nisbett)? Check. Statistical significance with t=1.99? Check. What could possibly go wrong???

I happened to read the article because E. J. sent it to me, but it’s not particularly bad. It’s far better than the himmicanes and hurricanes paper (which had obvious problems with data selection and analysis), or the ovulation and clothing paper (data coding problems and implausible effect sizes), or the work of Marc Hauser (who wouldn’t let people see his data), or Daryl Bem’s ESP paper (really bad work, actually I think people didn’t even realize how bad it was because they were distracted by the whole ESP thing), or the beauty and sex ratio paper (sample size literally about a factor of 100 too low to learn anything useful from the data).

I guess I’d put this “hungry lunch” paper in roughly the same category as embodied cognition or power pose: it could be true, or the opposite could be true (hunger could well reduce the desire to acquire of nonfood objects; remember that saying, “You can’t have your cake and eat it too”?). This particular study is too noisy and sloppy for anything much to be learned, but their hypotheses and conclusions are not ridiculous. I still wouldn’t call this good science—“not ridiculous” is a pretty low standard—but I’ve definitely seen worse.

And that’s the point. What we have here is regular, workaday, bread-and-butter pseudoscience. An imitation of the scientific discovery process that works on its own, week after week, month after month, in laboratories around the world, chasing noise around in circles and occasionally moving forward. And, don’t get me wrong, I’m not saying all this work is completely useless. As I’ve written on occasion, even noise can be useful in jogging our brains, getting us to think outside of our usual patterns. Remember, Philip K. Dick used the I Ching when he was writing! So I can well believe that researchers can garner useful insights out of mistaken analyses of noisy data.

What do I think should be done? I think researchers should publish everything, all their data, show all their comparisons and don’t single out what happens to have p less than .05 or whatever. And I guess if you really want to do this sort of study, follow the “50 shades of gray” template and follow up each of your findings with a preregistered replication. In this case it would’ve been really easy.