USA TODAY research reveals flaws in mass-killing data

When USA TODAY started investigating mass killings, it seemed a fairly straightforward thing to count: How many times have at least four people died at the hands of another in a single incident?

Yet marking the death toll of mass killings in America is anything but simple. It's hampered by the FBI's voluntary reporting system that gets it right a little more than half the time, and by advocacy groups who may count only incidents that support their cause, ignoring killings that don't involve a gun or did not get heavy media coverage.

Concentrating on just one type of mass killing — or only on those that get a lot of attention — may be worse than just using the FBI data, because it can skew public understanding and lead to ineffective policies, says Grant Duwe, a senior researcher with the Minnesota Department of Corrections, who has written a book on mass killings based on a data set he built covering the 1900s.

"Accurately accounting for mass killings — having a definition, sticking to the definition, trying to find all the incidents — that may seem somewhat pedantic but it's actually very important."

To get a more accurate count, USA TODAY began with 156 such incidents reported to the FBI from 2006-11. But after investigating each one and finding others missing, USA TODAY found the FBI data had an accuracy rate of just 61%, throwing doubt on conclusions that might be drawn from analyzing it.

For example, a mass killing in Samson and Kinston, Ala., in 2009 is not included in the FBI data. In that case, a man killed his mother, set her body on fire, then killed nine other people before he committed suicide at his former workplace.

In another incident, the FBI data included the deaths of two adults and three children in a Cleveland suburb in 2009, shot to death by a 28-year-old man. There was a drive-by shooting at that date and time, but no one was killed.

The FBI's data, known as the Supplemental Homicide Report, is considered the primary list of U.S. homicides by law enforcement agencies and academics studying violence. Most of the problem, researchers say, is because of mistakes made by the local police agencies who voluntarily submit their reports to the FBI.

The FBI acknowledges the data is flawed. In an effort to improve it, the FBI will start making downloads available directly from its website starting in 2014, said FBI spokesman Stephen G. Fischer Jr.

Depending on the agency, that means crime data could be updated daily, weekly or monthly, Fischer said. It also may mean mistakes can be caught more quickly. Currently, data is more than a year old when it's released (USA TODAY received preliminary 2011 data in February 2013).

"This will allow members of the media, academia, and the general public to download data sets and apply their own powerful trending and statistical analysis software," Fischer said.

Even with better data, special interest groups or unscrupulous academics can manipulate the numbers, just as with any other data set.

"If you have a cherry-picked list of cases, it's basically garbage in, garbage out," Duwe said. "And it does have important implications to additional research we do in terms of public policy.

Among the errors USA TODAY found:

Several mass killings were reported as unrelated single homicides.

At least a dozen crimes were mischaracterized as mass killings. In one case, several unrelated homicides a week apart were reported as a mass killing.

In nearly a dozen cases, USA TODAY — searching media reports and interviewing local law enforcement agencies — could find no record of a murder, even when the FBI data showed as many as seven killed. Among them: a quintuple murder in Newark in 2010. What actually happened: Police arrested two men in connection with the murder of five teens in 1978 — 32 years earlier.

Several cases handled by federal agencies were not included, including the 2009 Fort Hood massacre.

Florida and Native American reservations do not report homicides to the FBI. Nor did Nebraska or Washington, D.C., until 2009. USA TODAY found at least a dozen such cases. They were not counted in the error rate, however, since their absence is well-documented.

USA TODAY's data debunks common beliefs. For example, it shows that the number of mass killings has not increased in recent years; most occur among family members; and handguns, not assault weapons, are most commonly used.

"Hopefully, it'll raise folks' awareness," Duwe said. "There was this view that mass killings were so infrequent that their infrequency makes them unimportant. But the cost to society is enormous. It justifies a lot more attention into the topic than has been the case."