The database is a story of murders, suicides and suffering that's 10,000 data sets strong. It chronicles the deaths of inmates found in their cells; officers shot on duty; and incidents of drug abuse, domestic violence, hate crimes, gang activity and a litany of other wrongs. The data is murky and dispiriting both for what it depicts and what it does not.



Billed as the “Hall of Justice,” the repository designed by the Sunlight Foundation was released this month to spotlight the dearth of open criminal justice information available and the wealth of controversial issues surrounding data publication. In total, the transparency group said the gathering process required 18 months and support from multiple public and private institutions. The result is an exhaustive inventory of criminal justice data from all 50 states, which cumulatively, indicates a need for major policy reforms to bridge information gaps.



Sunlight’s Damian Ortellado, the research analyst who led the project, said this message grew more apparent as data collection progressed. Comparable data, and the standards that would create them, are nearly nonexistent throughout the U.S., he said. What’s more, most of the data that does exist isn’t digitally accessible — much of it being tied up in PDF documents.



“You are seeing a lot of this data scattered around the country in various localities from the national level, to state level, to county level, to even the city or neighborhood levels,” Ortellado said. “The goal of the project — with all of the post Ferguson energy in mind — was to show the extent of the problem through a consolidated and robust collection of data sets.”



Since that fatal shooting of 18-year-old Michael Brown by a Ferguson, Mo., police officer in 2014 — and subsequent controversies involving police use of force — data transparency reforms have commanded national attention as officials attempt to regain public trust. In 2015 the Obama administration’s Police Data Initiative committed 26 police departments to open data releases, and there are many independent efforts by law enforcement to produce similar results.



Still, what Americans see now is a fractured landscape, a picture Sunlight's new database paints well. There are cities like Seattle and states like California that are pioneering data transparency, while others don’t even show up in certain searches. For example, researchers found only 28 states had some type of data related to police use of force. And even within this sampling, much of the data was outdated, heavily redacted, or isolated to only a few localities.



“A lot of times with police use force incidents — or police shootings — those events are kind of restrained to internal data and internal affairs, and the officers are dealt with by their department,” Ortellado said.



Equally striking about the datasets was the disparity in privacy policies when publishing the data. The Florida Department of Corrections publishes the full name and cause of death for inmates who died in custody, while the California Department of Corrections & Rehabilitation redacts personally identifiable information on many in-custody deaths due to suicide and homicide for the deceased's family.



An office spokesman for California Attorney General Kamala Harris said the justice department, which manages the state’s law enforcement data site OpenJustice, redacts names and other personally identifiable information as part of a larger policy to adopt “responsible transparency.” Under the practice, the office has rolled out a series of data sets on its site that document arrest rates and violence against law enforcement, along with alleged perpetrators.



The office spokesman said they support measures like the Hall of Justice and other academic and research projects like it. For the Attorney General’s Office, the vision is to leverage transparency to enhance California’s criminal justice reporting via key performance indicators and interactive analytics tools for citizens — something they hope more jurisdictions embrace.



Moreover, in the wave of viral arrest phone videos and social media, the DOJ asserts that more data, not less, is the best way officers and criminal justice workers can show the whole picture of operations — as opposed to leaving it to the quick video snippets and online wordbytes.

At the forefront of this digital transformation for internal law enforcement data are govtech startups like Mark43 and SceneDoc, and Ortellado said it's likely more startups — or transparency companies like Socrata, which has partnered with the White House on its Police Data Initiative — arrive to assist with criminal justice data for citizens. As it stands now, Hall of Justice is meant to be a one-time research project, so consistency will require solutions from civic-minded entrepreneurs and jurisdictions themselves.



Along these lines, Ortellado commended localities like Portland, Ore., for its mapping and data work, and efforts in places like Dallas where leadership published incident reports in real time.



“I think with all the incidences happening lately, there's a strong desire to get out in front of these kinds of things, make data public facing and build community trust,” he said.

Looking for the latest gov tech news as it happens? Subscribe to GT newsletters.