[Content warning: this is a complicated analysis of something people care about a lot right now. I’m not confident in my analysis, the post comes to no clear conclusion and there are no easy answers about how to proceed. If I see this on Twitter with some headline about it DESTROYING somebody, I am going to be so mad.]

The New York Times says that It’s Time To Make Your Own Face Mask. But MacIntyre et al (2015) says it isn’t.

The surgical masks used in hospitals are made out of non-woven fabrics that are pretty different from anything you have at home. But in some developing countries, health care workers instead use masks made of normal cloth. Laboratory tests find that improvised cloth masks block 60 – 80% of virus particles. Respirators and real surgical masks block 95%+, but 60-80% still seems better than nothing. And most of the masks ordinary people wear in Asian countries are cloth, and they seem to do pretty well. So there’s some circumstantial evidence that these cloth masks might be helpful. Most experts in the early 2000s agreed that these masks were probably better than nothing. In 2015, an Australian team set out to prove it with a randomized controlled trial.

They went to a hospital in Vietnam and randomized workers there to a normal mask group, a cloth mask group, or a control group. Because it would have been unethical to tell the control group not to wear masks, they left the control group alone. Most control group workers did end up wearing masks sometimes, but less than the experimental groups did.

After a month, they counted how many infections each group had, for three different categories of infection. Here are the results:

Technically significant only in the ILI category, but later the authors do various post hoc adjustment for confounders and find it’s significant everywhere

For all three categories, people wearing the real surgical masks were the healthiest, the control group was in the middle, and people wearing the cloth masks were the sickest.

This shows real surgical masks work better than cloth masks. It’s a little bit unclear about how well cloth masks work. They do worse than the control group, but you could tell two stories about that. In one, cloth masks are worse than no mask at all. In the other, cloth masks have zero-to-slight-positive efficacy, but because some people in the control group were wearing real surgical masks some of the time, they did better than the cloth group overall. So it depends a lot what the control group was doing.

Unfortunately, the paper doesn’t give us all the data we want. It tells us that about 57% of both the surgical mask group and the cloth mask group wore masks regularly (defined as more than 70% of the time) but only 24% of the control group did. But there is no way of knowing whether the rest of the control group wore masks 69% of the time or 0% of the time.

A separate paragraph tells us that 37% of the control group used surgical masks, 8% cloth masks, and 53% used a combination of both. These numbers don’t make a lot of sense in the context of the last paragraph, so I’m going to assume they meant that on the infrequent occasions they did wear masks, those were the masks they used. But we don’t know if the compliant workers were disproportionately using cloth masks, disproportionately using medical masks, or both evenly. It’s hard to just eyeball these numbers and get a good sense for whether cloth masks really are worse than nothing.

But the authors themselves lean towards the hypothesis that that cloth masks are actively bad. First, because after some calculations I cannot quite follow, they find that the difference between surgical masks and cloth masks is so high that either the surgical masks are absurdly good, or the difference is being augmented by the cloth masks being actively bad. But nobody has previously found surgical masks to be absurdly good. The authors cite two previous studies of theirs which did include a no-mask control group; surgical masks did not significantly outperform nothing (they did show a trend towards doing so, and the studies were probably underpowered).

Second, because they compare the numbers from this study to numbers from those other two studies directly. They find the rate of infection in surgical mask users is not-significantly-different throughout the three studies, and the rate of infection in surgical mask users and no-mask controls was also not-significantly-different, and therefore surgical masks are the same as nothing and so probably the cloth masks are actively bad.

I am very unimpressed by this. First, you are really not supposed to compare things across multiple different studies. The authors protest that they did all three studies along pretty similar designs, but also admit they were different hospitals during different seasons. But second, almost no differences anywhere are significant, because all of these studies were at least a little underpowered. The current study found no significant difference between cloth masks and surgical masks in two of the three categories, even though the trend was in the expected direction. The other studies found no difference between wearing a medical mask and not wearing a medical mask, even though previous studies have suggested medical masks should work. They couldn’t even find any difference between wearing an N95 respirator and not wearing any protection at all. So when you need a chain of “x is not significantly different from y, which is not significantly different from z” in a bunch of studies that wouldn’t have been able to notice significant differences even if they existed, I stop believing it pretty quickly.

(In fact, I think you could use the same logic to draw the exact opposite conclusion. The cloth mask group in the current study didn’t have a significant difference from the surgical mask group in the other study, and the surgical mask group was no different from placebo, therefore cloth masks cannot have a negative effect. I find it hard to believe the authors missed this, so let me know if I am confused here.)

But MacIntyre et al take it seriously, and conclude:

The study suggests medical masks may be protective, but the magnitude of difference raises the possibility that cloth masks cause an increase in infection risk in HCWs. Further, the filtration of the medical mask used in this trial was poor, making extremely high efficacy of medical masks unlikely, particularly given the predominant pathogen was rhinovirus, which spreads by the airborne route. Given the obligations to HCW occupational health and safety, it is important to consider the potential risk of using cloth masks […] The physical properties of a cloth mask, reuse, the frequency and effectiveness of cleaning, and increased moisture retention, may potentially increase the infection risk for health care workers. The virus may survive on the surface of the face-masks, and modelling studies have quantified the contamination levels of masks. Self-contamination through repeated use and improper doffing is possible. For example, a contaminated cloth mask may transfer pathogen from the mask to the bare hands of the wearer. We also showed that filtration was extremely poor (almost 0%) for the cloth masks. Observations during SARS suggested double-masking and other practices increased the risk of infection because of moisture, liquid diffusion and pathogen retention. These effects may be associated with cloth masks.

Why am I focusing on this one weird study so much? Because it’s the only RCT of cloth face masks we have! Millions of people, egged on by top newspapers, are about to start wearing cloth face masks during a pandemic, when right now the authors of the only randomized trial on them conclude they’re probably net harmful. This should be really scary! Somebody with more experience and statistical knowledge than I have should be looking this over with a fine-toothed comb and trying to figure out what we should do.

Until then, should people stay away from cloth masks? I’m not sure, and this is so not a recommendation, but I lean toward no. The prior that they should work or at least be neutral is too high for a study this weak to convince me otherwise. More important, this study only examines incoming pathogens. Even if they are harmful for blocking incoming pathogens, there are still reasons to think they are helpful for blocking outgoing ones. If I had to hang out with a coronavirus patient for a while, and I had to choose between both of us wearing cloth masks, or neither, I would go with the masks. Only until we could get real surgical masks, which are much better. But I’d go with the cloth ones instead of nothing.

But right now that’s a gut judgment, and the evidence says I’m wrong. This is one of those times where people have to make a life-or-death decision in conditions of high uncertainty, and it really sucks.

[EDIT: Bolded a passage I think is important to make sure people don’t miss it]