A study just came out, Mortality in Puerto Rico after Hurricane Maria, by Nishant Kishore et al.:

Using a representative, stratified sample, we surveyed 3299 randomly chosen households across Puerto Rico to produce an independent estimate of all-cause mortality after the hurricane. Respondents were asked about displacement, infrastructure loss, and causes of death. We calculated excess deaths by comparing our estimated post-hurricane mortality rate with official rates for the same period in 2016. From the survey data, we estimated a mortality rate of 14.3 deaths (95% confidence interval [CI], 9.8 to 18.9) per 1000 persons from September 20 through December 31, 2017. . . . a 62% increase in the mortality rate as compared with the same period in 2016. However, this number is likely to be an underestimate because of survivor bias. The mortality rate remained high through the end of December 2017, and one third of the deaths were attributed to delayed or interrupted health care. Hurricane-related migration was substantial.

The 14.3 per 1000 is an annualized rate: it’s 14.3 deaths per 1000 people per year. The population of Puerto Rico is 3.4 million (or 3.3 million, or 3.7 million, depending on where you look it up), so 14.3 deaths per 1000 people is an estimated 49,000 deaths per year. They’re comparing that with a previous death rate of 8.8 per 1000 per year, which comes to 30,000 deaths per year. The difference—excess deaths—is 19,000 in a year, or 5,300 in the 102 days of the study, assuming a constant mortality rate throughout the year.

Here are some data they report from official death tallies:

It would help if they were to label the gray lines. I think death rates were going up from 2010-2016 but I’m not sure. Also, I have no idea why most years show a big jump in deaths from November to December. Is that real, or is is some data issue? The drop from November to December in 2017 (in contrast to the jump in all the other years) is consistent with under-reporting of deaths, and I guess it’s also consistent with people leaving the territory who would otherwise have died.

Anyway, from the above figure you can see that reported deaths did not dramatically increase from Sep-Dec 2016 to Sep-Dec 2017; even the months showing the biggest year-on-year trends show increases of less than 500 per month.

So the claim from the survey is that a lot of deaths went unreported. I don’t know how this works, exactly. I’m not saying the survey results are wrong, I’m just saying I don’t know enough about what was going on in Puerto Rico to know how to think about these survey responses.

Here are all the ages and months of deaths reported in the survey of 3299 households:

This graph is useful, not just in giving an estimate for the number of post-hurricane deaths but for showing the comparison: 18 deaths in the 263 days before the hurricane and 38 deaths in the 101 days after the hurricane. Indeed, this gives a raw estimate of relative death rate of (38/101)/(18/263) = 5.5.

A ratio of 5.5: that seems a bit too high! This makes me wonder about reporting bias.

On a separate matter, the the authors say that their survey is underestimating the number of deaths because there will be no response from households where everyone died (or a one-person household where that one person died).

I have some issues regarding survey design and weighting.

The design involves stratification (which makes sense) and clustering (which makes sense) and a procedure for getting replacement samples for abandoned homes. I’m not so sure that this last idea was so good, because the result will be to overrepresent areas with more abandoned homes. To get a sense of the importance of this bias, you’d want to know how often this happened in the data, and where. Also this replacement procedure involved “sampling from all surrounding visible houses” so I’m not sure what they did with apartment buildings.

The paper describes a two-stage weighting process, but then in the statistical analysis they don’t use their weights or account for clustering at all! Which makes me wonder why they were talking about the weights at all. They also get a standard error for the number of deaths using the Poisson distribution: That ain’t right! What you’re supposed to do is estimate the rate in each cluster and then use the cluster-level analysis to get your estimate and uncertainty. (Or fit a hierarchical model, but here I’m just talking about the classical approach.)

I also got an email from David Manheim who had some issues with the death rate comparisons. Manheim writes:

Figure S2 has the official death tolls, and makes me wonder how they justify their very high estimates and makes me wonder why they don’t discuss the time-shifting, given the much lower than usual death toll in November/December 2017. The survey methodology is also less than ideal for what they are doing. They could have used the same dataset that they used for the baseline in 2017, which was the number of official monthly deaths in Puerto Rico. Instead, the sampling they did has tons of other potential biases and issues. As one specific example, they used random resampling of missing households and those who refuse to consent. This is not horrible, but it creates a bias they don’t seem to adjust for. . . . the sampled median age was a decade older than the actual average – indicating what seems to be an obvious bias against working-age individuals, who were presumably less likely to be home when sampled, and shifting the age ranges presumably inflates the death tolls somewhat. Lastly, the normal counting of deaths from hurricanes is direct deaths. The original death toll report, which all of the press cites as misleading, reflects this number. The obvious comparisons to other storms is to that same number – direct deaths. The number they report is defensible, and as they cited, CDC recently recommended using it more generally, but they chose to estimate it for this particular storm, and not include comparisons to any other storm.

My overall take on this is that it’s a hard problem. It makes sense to do this sort of survey as a cross-check on official death tolls. But the uncertainties and possible biases in the estimates are so large that it’s hard to know what to do with the numbers.

Sometimes this can be difficult to capture in news reporting: the idea that this is a careful, high-quality study with summary numbers that are noisy. There’s a temptation to either dismiss the study entirely or to take its estimates as truth, but neither of those extremes is right.

It’s good news that all the data and code are publicly available so anyone can do alternative analyses.

Here’s the published paper, here’s the published supplementary material, and here’s the Github page with the data and code. The repository includes all the data except the gps locations, you get barrio instead. In particular, I’ve been told that the posted data include cluster identifiers so you should be able to analyze the survey respecting the design. The survey also includes potentially interesting data on questions about neighbors, so lots of things to explore for students or anyone else interested in going in depth here.

P.S. Rafael Irizarry, the statistician on this project, adds some comments:

I am convinced that the drop in Nov and Dec 2017 in monthly level data has to do with the fact that the government demographers are still catching up. I base this on the attached plot showing daily data. [I’m not sure if I have permission to share this plot, but I can describe it here. The reported deaths are approximately 100 per day in January 2017, then gradually decline to about 75 per day from March through October, then the hurricane lands, and deaths spike to about 120 per day for a couple weeks and then drop quickly to 100, then 75, then 50, then 25. — AG.] You can see an almost monotonic drop starting in mid-October, consistent with this being an incomplete dataset. If the rate remained constant at the rate we see in late September / early October, continuing through December, we end up with an estimate well within our confidence interval. The reason we have two datasets here is because we could not get the government to share the daily level data nor the monthly level data for 2017. The New York Times somehow got the daily data and shared it with us. The PR Institute of Statistics, an autonomous branch of the government, sent us the monthly data you see in the paper. But neither are official. Regarding apartments, I checked with other authors and they tell me that for apartments we had a random selection of floor, then unit in that floor. The larger ratio you note between the before-and-after rate is perhaps due in part to survival bias: the denominator is smaller than it should be. After applying the adjustment we describe, that ratio of rates goes down some. Our findings go beyond the estimation of the death count. For me, the most important takeaways were:

1- A substantial number of the post hurricane deaths were due to lack of access to medical care. Many more than due to trauma.

2- There is evidence that the death rate was high through December, not just right after the hurricane.

3- The survey that got us this information was completed in 3 weeks for less than $75K. Government agencies can run surveys like this, preferably larger ones, right after a disaster to get an idea of what is going on.

These are all interesting points. From a mathematical or statistical standpoint, the challenge arises because we are trying to estimate the frequency of a (fortunately) rare event from a general-population survey. Just think about it: They had to survey 3300 households to learn about 38 post-hurricane deaths. Whenever you’re in the position of needing to survey thousands of people to find out about only 38 events, you know you have a statistical challenge: any estimated total will be noisy, and estimates for subsets will be even worse. So I think Rafael is right that, if you’re going to do this sort of survey, you should look at all the information you can and not get stuck on any particular number. Policymakers should be combining what they learn from different sources.