I. Introduction

The impact of the U.S. government’s targeted killing policy on civilians and their property is especially controversial. Some argue that targeting---in particular, drone technology---reduces civilian casualties vis-à-vis other modalities of warfare, making targeted killings an attractive policy tool on this score. Some even argue that targeted killings are necessarily proportional, thus ensuring U.S. compliance with international law. But others argue that the number of civilian casualties caused by targeted killings is too high (and increasing) and that targetings inherently violate international law. Some attack the data itself, arguing that there are deep inaccuracies in civilian casualty estimates, making it impossible for the U.S. and others to drive improvement and to identify and assist civilian victims. Others argue that collateral deaths are problematic in that they harmfully feed back into other policy vectors, namely undermining the credibility of U.S. counterterrorism programs and harming the United States’ relationship with other countries, friend or foe.

II. Data

Scott Shane reported in August 2011 that, according to American officials, the U.S. Government had killed more than 2,000 militants and only about 50 noncombatants since 2001. According to Shane, John O. Brennan initially said that, in the year preceding June 2011, “there ha[d]n’t been a single collateral death" resulting from USG drone strikes. Yet, that claim gave rise to "deep skepticism from many independent experts," and Brennan later revised his statement stating instead that the USG could not confirm any such collateral deaths.

Ben Emmerson, the U.N. Special Rapporteur on Human Rights and Counterterrorism, reported in March 2013 that he had been informed that USG drone strikes in Pakistan routinely inflicted civilian casualties. Specifically, Emmerson's report stated that he had been "informed that according to statistics compiled by [Pakistan's] Ministry of Foreign Affairs, there have been at least 330 drone strikes on the territory of Pakistan since 2004" and that "the total number of deaths caused by drone strikes was at least 2,200 that in addition at least 600 people had suffered serious injuries."

According to Philip Alston, three entities collect and analyze the bulk of existing non-governmental data about civilian casualties resulting from the U.S. government’s targeted killing policy: (1) the New America Foundation; (2) the Long War Journal; and (3) the Bureau of Investigative Journalism. Their estimates of the civilian casualty rate range widely, from something like 1% to nearly 35% of all deaths caused by targetings.

The New America Foundation, which collects data from different media sources, estimates a civilian casualty rate of approximately 17% between 2004 and 2011 and of 6% since 2010. The Long War Journal, which likewise culls data from media reports, estimates a non-militant casualty rate of slightly less than 10%. Like the New America Foundation, the Long War Journal finds that the civilian casualty rate has declined in recent years, to something like 8.5%. The Bureau of Investigative Journalism, which seems to base its data on media sources and other existing databases, estimates a considerably higher civilian casualty rate of 13-34%. Various other sources, including official Pakistani data, the New York Times, the Campaign for Innocent Victims in Conflict (CIVIC), and Reuters, have estimated a range of civilian casualty rates, most of which fall within the spread set out by the three main databases discussed above.

Some have criticized these studies. For instance, Kenneth Anderson has stated that "groups attempting to estimate numbers do not seek to disaggregate" the data regarding conventional combat from that related to targeted killing as such and "thus total numbers killed might well go up or down significantly as a function of conventional combat in Afghanistan, not as a result of counterterrorism operations.” Others, according to Jacob Beswick, "question[] the validity of New America Foundation’s work as it depends entirely on news sources.” Ben Wittes has stated that estimates of civilian casualty rates are subject to errors because reporting from the relevant regions is incredibly dangerous and thus "estimates are largely based on reports in anonymously-sourced media stories that rely on Pakistani Army claims that 'militants' were killed," which, at best, "leaves the estimators at the mercy of the errors, the self-serving accounting, and the agendas of the Pakistani Army sources." Likewise, Wittes states, estimators do not agree about what they are counting, and thus any data report is heavily influenced by the "normative, factual, and legal assumptions an estimator brings to the table." Scott Shane, too, has reported the disparities between the data reported by the U.S. government, the Pakistani government, the media, and other sources.

II. a. Data Collecting Organizations

i. New America Foundation (NAF):

Under the direction of Peter Bergen, Director of NAF’s National Security Studies Program, a team of researchers at the Washington DC-based think tank compiles data on drone strike casualties in Pakistan from 2004 to the present. In the organization’s words,

This database seeks to. . .provid[e] as much information as possible about the covert U.S. drone program in Pakistan in the absence of any such transparency on the part of the American government. The data was collected from credible news reports and is presented here with the relevant sources.

NAF also collects data on drone and air strike deaths in Yemen from 2002 to the present.

To be precise, NAF does not collect data on drone strike deaths directly, but on media reporting about drone strike deaths. It does not do independent fact gathering, nor does it seek to verify the accuracy of the press reports it uses. It simply counts deaths reported in media sources it considers “credible.” Its methodology is, therefore, entirely predictable, replicable, and has the benefit of consistency. It is also entirely derivative and only as good as the sometimes-flawed reporting that lies beneath it.

As the organization explains, the sources it uses to build its database include:

[T]he three major international wire services (Associated Press, Reuters, Agence France Presse), the leading Pakistani newspapers (Dawn, Express Times, The News, The Daily Times), leading South Asian and Middle Eastern TV networks (Geo TV and Al Jazeera), and Western media outlets with extensive reporting capabilities in Pakistan (CNN, New York Times, Washington Post, LA Times, BBC, The Guardian, Telegraph).

It separates casualties into three categories: “militant,” “civilian,” and “unknown.” It labels the dead as militants when “two or more news reports label the dead as ‘militants,’ while others call them ‘people’ or some other neutral term.” It categorizes them as civilians when “two or more media outlets explicitly refer to the dead as ‘civilians,’ ‘women,’ or children’.” And it labels the casualties as unknowns when “a majority of reports do not refer to the dead as ‘civilians,’ ‘women,’ or ‘children,’ but one media outlet does” or when “the various media reports are [too] contradictory” to make a judgment.

NAF resolves any conflicts it comes across in its counting using the following standard: When “a media report cites ‘some’ ‘civilians/women/children’ but does not specify how many, and no other media sources provide a specific total, [it] report[s] one third of the total victims referenced in that source as ‘civilians’ or ‘unknowns’.”

As of July 22 2013, NAF’s casualty counts for Pakistan stand at: 258 to 307 civilians killed, 1,585 to 2,733 militants killed, and 196 to 330 unknown killed. The total number of people killed is 2,039 to 3,370. The rate of the civilian deaths, in other words, ranges between eight and fifteen percent.

ii. Long War Journal (LWJ):

The LWJ is a project of the Foundation for the Defense of Democracies, also a Washington DC think tank. Its managing editor, Bill Roggio, leads the effort to document drone strike casualties in Pakistan from 2006 to the present. Roggio and his team classify the dead in one of only two categories: “Taliban/Al Qaeda casualties” or “civilian casualties.”

LWJ also aggregates data on casualties from airstrikes in Yemen from 2002 to the present, and it classifies the dead using a similar either-or form: “Al Qaeda in the Arabian Peninsula Casualties” or “civilians.”

While the other organizations working in this space have substantial sections explaining their methodologies, LWJ does not. It simply says on its website: “The data is obtained from press reports from the Pakistani press (Daily Times, Dawn, Geo News, The News, and other outlets), as well as wire reports (AFP, Reuters, etc.), as well as reporting from The Long War Journal.” Whenever a strike occurs, Roggio publishes a post (such as this one about a recent drone strike in North Waziristan on July 2, 2013), and updates a number of charts on the organization’s website.

In a January 2010 article published on LWJ’s website, Roggio and Alexander Mayer wrote: “[I]t is possible to get a rough estimate of civilian casualties by adding up the number of civilians reported killed from the media accounts of each attack.”

As of July 22 2013, LWJ claims “2,526 leaders and operatives from Taliban, Al Qaeda, and allied extremist groups” and “153 civilians” have been killed in Pakistan since 2006. This yields a civilian death rate of nearly six percent.

iii. The Bureau of Investigative Journalism (BIJ):

BIJ describes itself as an “independent not-for-profit organization” whose aim is to “pursue and encourage journalism in the public interest.” It is based out of City University in London, and Chris Woods directed the Covert Drone War investigation until recently, which Alice Ross now leads.

BIJ documents drone strike casualties in Pakistan from 2004 onwards, and covert action casualties (in other words, victims of drone strikes as well as “airstrikes, missile attacks and ground operations”) in Yemen and in Somalia.

Unlike NAF and LWJ, whose focus is on counting militant deaths, BIJ places a focus on identifying non-militant deaths---which it categorizes as “civilians reported killed” with a subcategory of “children reported killed.” According to its methodology:

For each reported US attack, the Bureau seeks to identify the time, location and likely target, and to present as clear a description as possible of what took place during the event. We also seek to identify the numbers of those reportedly killed and injured, and to ascertain when possible whether they were alleged militants, or civilians. Wherever possible, we include other information on casualties, such as name, gender, age, tribal affiliation and other identifying aspects.

BIJ relies on a greater number of international and local media sources than do NAF or LWJ to arrive at its numbers. It also uses a more complex methodology than do the other two groups, because it incorporates---in addition to the press reports---a dynamic array of sources in its counting: information for example, from three field investigations it has conducted in Pakistan, from WikiLeaks, and from publicly available documents such as lawsuits.

When BIJ runs into discrepancies in its counting, it resolves them in the following manner:

Where contradictory accounts occur, we strive to speak with particular journalists and sources about their reports to clarify discrepancies. But where these discrepancies remain we include the contrasting accounts in the datasets’ narratives of each strike.

. . .

Where the reporting is vague but appears to indicate civilian casualties, we will include the line “Possible reported civilian casualties” in that strike’s casualty figures in the Timeline. . . .Where the reporting is more specific, but conflicts with other reports or is from a single source, we use the formula 0-X in our count of civilian deaths, with X referring to the highest reported number of civilian casualties. . . .This ensures that the minimum total number of reported civilian deaths is unchanged but the maximum total incorporates these possible civilian deaths.

As of this writing, BIJ’s number of civilians killed in Pakistan since 2004 runs from 411 to 890 (the number of children killed is 167 to 197). The total killed is between 2,566 and 3,570. This leaves a civilian death rate that ranges greatly---between as low as twelve percent and as high as thirty-five percent.

iv. Columbia Law School’s Human Rights Clinic (CHRC):

Columbia’s Human Rights Clinic published a study entitled Counting Drone Strike Deaths in October 2012. Its primary author is Chantal Grut, then an LLM candidate, who was supervised by Naureen Shah, then the associate director of the Counterterrorism and Human Rights Project at Columbia.

The stated aim of the report is “to thoroughly examine the data and methodology of the three tracking organizations.” The study does not locate or use any sources of its own, but instead uses the data on which NAF, LWJ, and BIJ rely in order to arrive at its own independent recount of casualties for drone strikes in Pakistan in the year 2011. The study seeks to understand the reasons for the discrepancies between the tracking organizations’ estimates, and to explain the limitations of media reports as a metric to count drone strike casualties.

In its recount, “where a hyperlink was broken and appeared to be the source of an organization’s upper or lower casualty figure, [CHRC] tried to re-source the article,” and re-categorized every death as a militant or civilian casualty.

The Human Rights Clinic bases its classifications on making a “strong,” “medium,” or “weak” identification of each death. The organization defines a strong identification as one in which “the deceased are individually identified by name, and/or where the reported identification of the deceased is corroborated by an independent investigation.” A medium identification is one in which “there are multiple original sources for the identification of the dead. For example, both anonymous officials and a local resident.” And a weak identification is one in which “there is only one source for the identification. For this purpose, [the study] treat[s] multiple anonymous officials as one source, and plural unnamed residents as one source.”

When confronted with issues in the data, the organization takes “a lower figure of 0” in “cases where [it] had concerns with the media sources,” and “[i]n cases where there are conflicting reports about the identity of the individuals killed as either militants or civilians, [its] figures reflect both possibilities.” When pushed for further clarification, Grut responded in an email saying:

“[I]n cases where the identity of a person killed wasn't clearly reported or was disputed, we counted it as 0-1 civilians killed and 0-1 militants killed, which I believe is the most accurate reflection of the possibilities. So for example for a strike on Jan 7, we counted 0-6 civilians killed and 0-6 militants, but a total killed of 4-6 (rather than 0-12).”

The report found that:

First, despite the strong efforts of the tracking organizations, their estimates of civilian casualties are hampered methodologically and practically. Two of the organizations [NAF and LWJ], according to our independent review of the media sources available, significantly and consistently underestimated the potential number of civilians killed in Pakistan during the year 2011. Second, while some of the flaws we identify can be fixed, others are inherent to the process. . . .

The Human Rights Clinic concludes that BIJ’s numbers were the most accurate and closest to its own estimate. It found that the number of alleged militants killed in Pakistan in 2011 was between 330 and 575, the number of alleged civilians killed was between 72 and 155, and the number of total killed ran from 456 to 661. This produces a civilian casualty rate that, like BIJ’s, also ranges hugely---from eleven percent to thirty-four percent.

v. Stanford Law School and NYU Law School:

The International Human Rights and Conflict Resolution Clinic at Stanford and the Global Justice Clinic at NYU released a report entitled Living Under Drones in September 2012. The organizations were contacted by Reprieve, a UK-based human rights organization, about conducting this investigation.

Like the Columbia Human Rights Clinic study, this one does not seek to collect data on drone strike deaths itself. It does not even seek, as that study does, to do a recount of the other organizations’ data. Instead, it describes in considerable detail a variety of factors that can lead to the conflicting numbers of drone strike casualties. For example, the study explores the limitations of media sources, and explains how news agencies can report conflicting information about drone strikes:

Those who work for major news outlets and wire services tend to spend more time embedded with military and intelligence officials and are thus more likely to report “official” accounts. Those who are not escorted into FATA by the military rely more on locals and stringers. The result is that different journalists with different contacts get different stories, make different decisions about who to trust, and frequently end up publishing conflicting accounts of each strike.

The report also outlines other considerations that may affect the estimates, such as limited first-hand knowledge of the strikes and unreported strikes. Its most significant addition to the discussion, however, comes from the two investigations in Pakistan that underlie it. The research team conducted

[O]ver 130 detailed interviews with victims and witnesses of drone activity, their family members, current and former Pakistani government officials, representatives from five major Pakistani political parties, subject matter experts, lawyers, medical professionals, development and humanitarian workers, members of civil society, academics, and journalists.

. . .

Investigations included interviews with 69 individuals (‘experiential victims’) who were witnesses to drone strikes or surveillance, victims of strikes, or family members of victims from North Waziristan.

The Stanford-NYU group concludes with the Columbia Human Rights Clinic: the Bureau of Investigative Journalism’s “data currently constitute the most reliable available source,” compared to the numbers of the New America Foundation or the Long War Journal.

III. Problems Associated With Civilian Casualty Data

Despite the proliferation of data about the collateral damage caused by targeted killings, questions about the soundness of that data remain. First, it is difficult to distinguish militants from non-militants, particularly where non-militants live alongside civilians and do not wear uniforms. Second, civilian casualty data is imperfect, as media access to locations where targeting occurs is limited. Third, sources---on both sides of the issue---have incentives to manipulate the data for strategic and political gain. Even non-governmental sources might have substantive aims that may undermine the neutrality of their data. According to Stephen Holmes and others, these inaccuracies are problematic because they make it impossible for the U.S. government and others to measure the effects of targeted killings, to drive improvement in the numbers, and to identify and assist civilians harmed in targeting strikes.

Moreover, civilian casualties themselves---irrespective of data collection problems---are seen as problematic, not only morally and legally, but also strategically. For instance, Brian Glyn Williams claims that civilian deaths foment anti-American sentiment and drive the populace to support insurgencies. Likewise, some argue that drones increase the temptation for the U.S. to engage in war, as they reduce the personal risk to U.S. troops. But authors like Kenneth Anderson vigorously dispute this proposition.

III. a. Reconciling the Different Estimates

Because the New America Foundation, the Long War Journal, the Bureau of Investigative Journalism, and the Columbia Law School Human Rights Clinic all have slightly different priorities in the way they count the dead, each organization comes to a different conclusion about who---and how many people---are being killed.

To illustrate the differences between each organization’s estimates, the following methodology is used:

1) The numbers of U.S. drone strike casualties in Pakistan in 2011 from each report is consulted.

2) NAF’s are used and when an organization didn’t have an estimate that corresponded to that category, it was left blank.

3) Columbia’s Human Rights Clinic also did this comparison in their study, but the data below are slightly different from those that it reports. The reason is perhaps that the numbers each organization puts forth now for 2011 are used, and some of the groups likely updated their data in the time since Columbia’s report was published---which noted some errors in their data.

Number of Deaths from U.S. Drone Strikes in Pakistan in 2011

NAF LWJ BIJ CHRC Militant 303 - 502 405 330 - 575 Civilian 57 - 65 30 52 - 146 72 - 155 Unknown 32 - 37 Total 392 - 604 435 447 - 660 456 - 661 Civilian Casualty Death Rate 9% - 17% 7% 8% - 33% 11% - 34%

The New America Foundation and the Bureau of Investigative Journalism count deaths from drone strikes from 2004 onwards. Unfortunately, the Long War Journal doesn’t have data available for 2004 and 2005, NAF and BIJ’s numbers are compared for reasons of consistency.

Number of Deaths from U.S. Drone Strikes in Pakistan from 2004-Present

NAF BIJ Militant 1,585 – 2,733 Civilian 258 - 307 411 - 890 Unknown 196 - 330 Total 2,039 – 3,370 2,566 – 3,570 Civilian Casualty Death Rate 8% - 15% 12% - 35%

The three organizations also aggregate data for casualties in Yemen from 2002 to the present, although these numbers are much more difficult to compare. NAF counts “drone and air strike deaths,” while LWJ counts casualties from “airstrikes” (a term it also uses to refer to drone strike deaths in Pakistan, though it is not clear whether LWJ is counting drone and airstrike deaths in this context). And BIJ’s numbers reflect “confirmed” and “possible” deaths from “airstrikes, missile attacks and ground operations” in Yemen, because of the uncertainty and speculation over whether the Yemeni government or the US has carried out many of these attacks. As a result, a direct comparison is not possible here, but broadly speaking, NAF reports between 557 and 760 militant deaths and a total of 596 and 832 total casualties. LWJ puts AQAP deaths at 349 and civilian deaths at 82. And BIJ’s estimates are between 15 and 52 civilians killed and 239 and 349 total deaths as a result of “confirmed” drone strikes---and between 23 and 48 civilian casualties and 283 and 456 total casualties as a result of “possible” US drone strikes.

At one level, these numbers seem irreconcilable and demonstrate exactly why this debate is so contentious. The New America Foundation estimates that, at the high-end, seventeen percent of all deaths in 2011 in Pakistan were civilian deaths. At the other extreme, the Bureau of Investigative Journalism and Columbia’s Human Rights Clinic put their high-end estimates at 33 and 34 percent, respectively---a number that is twice as high as NAF’s. That number is also nearly four times NAF’s low-end estimate of nine percent.

Discrepancies like this allow those who participate in this debate to see in the data what they want to see. Proponents of the targeted killing program point to the precision and accuracy of drones in waging war, and NAF and LWJ’s low-end estimates of eight and six percent, respectively, corroborate their claims. Those who are opposed to the program on moral and humanitarian grounds can reasonably point to the BIJ and CHRC’s high-end estimates to substantiate their assertions. Neither side is acting unreasonably.

Reconciling these various numbers, however, is possible. The high-end estimates of NAF and LWJ are broadly consistent with the low-end estimates of BIJ and CHRC---so one could take the zone of overlap as representing an evocative common ground. For instance, NAF’s high-end estimate for Pakistan from 2004 to the present is fifteen percent, while BIJ’s low-range estimate for the same period is twelve percent. In another example, in Pakistan in 2011, BIJ and CHRC estimate as low as eight and eleven percent civilian casualties, respectively, while NAF estimates as high as seventeen percent. Dropping the outlier for that period (the Long War Journal), one could think of the civilian casualty rate as likely ranging from eight to seventeen percent.

IV. Despite such problems, targeted killings may have some overriding advantages.

In particular, Professor Jack Goldsmith explains, conducting targeted killings via drone strikes is likely to impose a lower ratio of combatant to civilian deaths, as compared to other tactics. As drone technology has improved over time, Bill Roggio and Alexander Mayer argue, the number of civilian casualties caused by the U.S. government’s targeting policy has declined. Likewise, Peter Bergen & Katherine Tiedemann assert that the ratio of civilian casualties to militants killed has fallen as well. Consequently, Michael W. Lewis, Dakota S. Rudesill, Gabriella Blum, and Laurie R. Blank, among others, put forth the argument that the United States is better able to comply with the international law requirement of “proportionality” by using drone technology for targeted killings. And Toren G. Evers-Mushovic and Michael Hughes have even argued that, targeted killings are inherently proportional, thus perhaps making the use of drones for targeting imperative.

But, others, like the non-profit humanitarian organization Civilians in Armed Conflict (formerly known as CIVIC), disagree, instead claiming that the U.S. drone program is operated in a way that actually increases the risk of harm to civilians.



V. Further Reading