Earlier this month, a police officer shot and killed an unarmed black teenager, Michael Brown, in Ferguson, Missouri. The shooting and the response have reignited concerns about racial profiling, police brutality and police militarization. The incident has also drawn attention to a remarkable lack of knowledge about a seemingly basic fact: how often people are killed by the police.

Some reporting has put forward one of the only figures available: the approximately 400 “justifiable police homicides” each year since 2008, according to the FBI’s annual Supplementary Homicide Report (SHR). That data point has appeared with heavy caveats in a string of media reports, including in USA Today, the St. Louis Post-Dispatch and the Washington Post.

The statistic might seem solid at first glance. The Centers for Disease Control and Prevention and the Bureau of Justice Statistics — independently of the FBI — also estimate the number of police homicides per year at around 400.

But these estimates can be wrong. Efforts to keep track of “justifiable police homicides” are beset by systemic problems. “Nobody that knows anything about the SHR puts credence in the numbers that they call ‘justifiable homicides,’” when used as a proxy for police killings, said David Klinger, an associate professor of criminology and criminal justice at the University of Missouri who specializes in policing and the use of deadly force. And there’s no governmental effort at all to record the number of unjustifiable homicides by police. If Brown’s homicide is found to be unjustifiable, it won’t show up in these statistics.

The FBI’s Uniform Crime Reporting (UCR) program, which compiles the SHR, relies on voluntary involvement of state and local police agencies — a fact that may raise some questions about the integrity of the data. But many UCR numbers are considered fairly reliable, particularly fatal crimes like murder and manslaughter, according to Richard Rosenfeld, also a professor of criminology and criminal justice at the University of Missouri, who studies crime statistics.

The simple UCR database does have plenty of limitations. For one, the data has holes in it: Illinois, for example, changed its reporting guidelines in 2010 to comply with UCR methodology, and data is only available prior to 2010 for the state’s largest cities. And then there are cases like Dallas, which in 2007 quietly redefined what it considers an aggravated assault, lowering its violent crime figures in the database. Such discrepancies are a major problem for UCR reporting of non-fatal crimes.

But the bigger issue is that the basic UCR reports don’t include any information on victims or offenders. That data is provided to the FBI via a separate form, also called the Supplementary Homicide Report (SHR). The circumstances of each death are supposed to be recorded here, with classifications like “gangland killings” (code 46), “lovers triangles” (code 40), and “felon killed by police” (code 81).

The “400 justifiable police killings” figure comes from these SHR forms. This is a problem, for four main reasons:

Fewer local police agencies report SHR data than report standard UCR data. “Felon killed by police” refers narrowly to justifiable police homicides, and “unjustifiable homicide by police” is not a classification. This means it’s difficult to combine unjustifiable police homicides — which could be listed as crimes elsewhere in the database — with “justifiable” police homicides. If the legality of a police homicide is in question, it may not be reported to the FBI SHR until the investigation is resolved. If the investigation concludes in a new reporting year, the old SHR data may not be updated, regardless of whether the killing was found to be justifiable or not. Criminology professor Geoff Alpert of the University of South Carolina, an expert on police violence, said he has “never seen a department go back and audit their numbers and fix them.” (In a statement provided in response to emailed questions, the FBI confirmed that it generally does not reopen master data files to add or correct reports.) Killings in federal jurisdictions, such as federal prisons or military bases, are not included in the database.

The statement provided by an FBI spokesman, Stephen G. Fischer Jr., noted that the UCR and SHR databases aren’t quite comparable, though “agencies consistently report homicide data both on the [UCR] and in our more detailed SHR data.” Given these problems, the SHR’s “justifiable police homicide” number is not a useful approximation of how many people are killed by the police.

One reasonably reliable statistic we do have is the number of Americans who die in homicides each year. This data can be found in the Centers for Disease Control and Prevention’s National Vital Statistics System (NVSS) and is based on death certificates. The NVSS includes the cause of death classifications “homicide” and homicide by “legal intervention”: the geneal equivalent of the SHR’s “justifiable police homicides” number.

But the “legal intervention” deaths — an average of 406 per year from 1999 to 2011 — are not useful on their own since many police homicides are misclassified simply because the coroner’s report does not mention police involvement. This count, along with the FBI’s estimate and an independent count of “arrest-related homicides” by the Bureau of Justice Statistics from 2003 to 2009 complete the trio of official-sounding but incomplete police homicide stats.

The total number of homicides in the NVSS is considered reliable, and we can use that data to estimate how many homicides are missing from the SHR database. The chart below shows total killings, including “legal interventions,” complied by the NVSS from 1999 to 2011. We can compare this to the number of homicide victims (including those of “justifiable police homicides”) with entries in the raw FBI Supplementary Homicide Report data.

During this span, the NVSS database reported an average of 18,011 homicide deaths per year. The raw SHR data contained an average of 15,024 victims. (The spike in 2001 is due to 2,823 homicides associated with the Sept. 11 terrorist attacks.) That’s a roughly 3,000-person gap.

Some of those 3,000 deaths are police homicides, justifiable and unjustifiable — there’s no way of knowing how many. They also include other homicides that are not reported to the SHR but which have nothing to do with police involvement — for example, killings that occur in federal jurisdictions.

It’s likely there are homicides recorded in the SHR that should be attributed to police as “justifiable” but aren’t. And, as I mentioned earlier, there’s an unknown number of unjustifiable police homicides that aren’t marked with any evidence of police involvement. Account for all that, and you would have the true number of police homicides each year.

It’s more than 400.

This story has been updated to include comment from the FBI.

CLARIFICATION (Aug. 20, 1:15 p.m.): We’ve added a note to clarify that the “legal intervention” homicides tracked by the National Vital Statistics System include legal executions.