I understand the above quote completely. Life would be so much simpler if my work was just reviewed by my personal friends and by people whose careers are tied to mine. Sure, they’d point out problems, but they’d do it in a nice way, quietly. They’d understand that any mistakes I made would never have any major impact on our conclusions.

OK, not really. Actually I want my work reviewed by as many people as possibles. Friends and colleagues, yes. But strangers also. The sooner the better.

But I understand that lots of people want the review process restricted. That’s the whole principle of journals like Perspectives on Psychological Science or the Proceedings of the National Academy of Sciences: if you’re well connected, you can publish pretty much whatever you want. And some of these folks get pretty hot under the collar if you dare to question the work they have published and promoted.

It’s a pretty sweet gig. Before publication, you say the work is a fragile butterfly that can’t be exposed to the world or it might get crushed! It’s only safe to be seen by friends peers, who can in secret give it the guild seal of approval. The data need to be kept secret too. Everything: secret secret secret, like the deliberations before choosing the next pope. And, after publication, the paper is Published! so it can’t be questioned—if you find any flaws, it’s your responsibility to Prove that these flaws materially affect the conclusions, and you can’t ever be so mean as to suggest any lack of competence or care on the part of the researcher or the journal. Also, once it’s published, no need to share the data—who are you, annoying second-stringer, to keep bugging us about our data?? We’re important people, and we’re busy on our next project, when we’re not promoting the current work on NPR.

In this case, there’s no PNAS, but there is the willing news media. But not all the media are willing to play the game anymore.

The source of the above quote

It comes from the Buzzfeed article by Stephanie Lee that I mentioned last night.

Lee’s article recounted the controversies involved in with the Santa Clara and Los Angeles coronavirus prevalence studies that we’ve been discussing lately.

My favorite part was this quote from Neeraj Sood, a professor of public policy at the University of Southern California and one of the authors of the two controversial studies:

Sood said he plans to eventually post a paper online, but only once it has been peer-reviewed and approved for publication. “I don’t want ‘crowd peer review’ or whatever you want to call it,” he said. “It’s just too burdensome and I’d rather have a more formal peer review process.”

When I say this quote is my favorite part of Lee’s article, I don’t mean that I agree with Sood! Rather, it’s a great line because it reveals in distilled form a big problem with modern competitive science.

“It’s the best science can do.”

Before going on, let me emphasize that I’m not trying to paint Sood as a bad guy. I’m using his name because it’s his quote, and he can take responsibility for what he says (as can I for my own public statements), but my point here is that the attitude in his quote is, unfortunately, all too common in science, or in some corners of science. I’m amused because of how it was stated, but really it bothers me, hence this post.

Now for the background.

Sood and about 15 other people did two serious studies in a short time, sampling and covid-testing over 4000 people in two California counties. They’ve released none of their raw data or code. For one of the studies, they released a report with some summary data; for the others, bupkis. They did, however, make a bunch of claims in the news media. For example, Sood’s coauthor John Ioannidis was quoted in the New York Times as saying of their study, “It’s not perfect, but it’s the best science can do.”

That ain’t cool. They made mistakes in their analysis: that’s understandable given that (a) they were in a hurry, (b) they didn’t have any experts in sample surveys or statistics on their team. Nobody’s perfect. But it’s not “the best science can do.” It’s the best that a bunch of doctors and medical students can do when they don’t have time to talk with statistics experts.

What’s the point of describing hasty work as “the best science can do”? How hard would it be for him to say, “Yes, we made some mistakes, but what we found is consistent with our existing understanding, and we hope to do better in the future”?

But as long as news reporters will take statements such as “it’s the best science can do” at face value, I guess some scientists will say this sort of thing.

I have no problem with them going straight to the news media

I have no problem with Ioannidis et al. going to the news media. They have done work that they and many others have good reason to believe is influential. It’s policy relevant, and it’s relevant right now. Go on NPR, go on Fox News, go on Joe Rogan, go on Statistical Modeling, Causal Inference, and Social Science—hit all the major news media. I’m not one of those people who says, “What do we want? Evidence-based science. When do we want it? After peer review.” If you really think it’s important, get it out there right away.

But I am annoyed at them hyping it.

If that Santa Clara study was really “the best science can do,” then what would you call a version of the Santa Clara study that did the statistics right? The really best science can do? The really really best science can do?

It’s like in the Olympics: if the first gymnast to go out on the mat does some great moves but also trips a few times but you give her a 10 out of 10 just because, then what do you do when Simone Biles comes out? Give her a 12?

I’m also annoyed that they didn’t share their data. I can see there might be confidentiality restrictions, but they could do something here. For example, in the Times article, Ioannidis says, “We asked for symptoms recently and in the last few months, and were very careful with our adjustments. We did a very lengthy set of analyses.” But none of that is in their report! He adds, “That data will be published in a forthcoming appendix.” That’s good. But why wait? In the Los Angeles study, they not only didn’t share their data, they didn’t even share their report!

“Crowd peer review” is too “burdensome” and they couldn’t put in the effort to share their data or a report of their methods, but they were able to supply “B-roll and photos from the April 10-11, 2020 antibody testing.” Good they have their priorities in order!

Here’s what I wrote earlier today, in response to a commenter who wrote that those researchers’ “treatment of significant uncertainties contrasts with basics tenets of the scientific method”:

I disagree. The three biggest concerns were false positives, nonrepresentative sampling, and selection bias. They screwed up on their analyses of all three, but they tried to account for false positives (they just used too crude and approach) and they tried to account for nonrepresentative sampling (but poststratification is hard, and it’s not covered in textbooks). They punted on selection bias, so there’s that. I feel like the biggest malpractices in the paper were: (a) not consulting with sampling or statistics experts, (b) not addressing selection bias (for example, by looking at the responses to questions on symptoms and comorbidity), and (c) overstating their certainty in their claims (which they’re still doing). But, still, a big problem is that: 1. Statistics is hard. 2. Scientists are taught that statistics is easy. 3. M.D.’s are treated like gods. Put this together, and it’s a recipe for wrong statistical analyses taken too seriously. But, that all said, their substantive conclusions could be correct. It’s just hard to tell from the data.

Sood’s a professor of public policy, not an M.D., so point #3 does not apply, but I think my other comments hold.

Back to the peer-review thing

OK, now to that quote:

“I don’t want ‘crowd peer review’ or whatever you want to call it,” he said. “It’s just too burdensome and I’d rather have a more formal peer review process.”

What an odd thing to say. “Crowd peer review” isn’t burdensome at all! Just put your data and code up on Github and walk away. You’re done!

Why would you want only three people reviewing your paper (that’s the formal peer review process), if you could get the free services of hundreds of people? This is important stuff; you want to get it right.

As Jane Jacobs says, you want more eyes on the street. Productive science is a bustling city neighborhood, not a sterile gated community.

Conclusion

I’ll end it with this quote of mine from the Buzzfeed article: “The fact that they made mistakes in their data analysis does not mean that their substantive conclusions are wrong (or that they’re right). It just means that there is more uncertainty than was conveyed in the reports and public statements.”

It’s about the science, not about good guys and bad guys. “Crowd peer review” should help. I have no desire for these researchers to be proved right or wrong; I’d just like us all to move forward as fast as possible.