Each week in Chicago begins with a grim reckoning: reporters count up and record the victims of the weekend’s violence. Over Labor Day weekend, 29 people were shot and six killed. A month before, 75 shot, 13 killed. A year earlier, 42 wounded, nine killed.

In the last few years, crime in the city has skyrocketed. But the drumbeat of headlines has obscured the fact that Chicago wasn’t always like this.

Crime and Punishment in the Windy City

At the end of 2013, shootings and murders in Chicago had hit a near-record low, with just 421 murders and 2,100 nonfatal shootings, according to the University of Chicago. That’s not good—controlling for population, the murder rate in 2013 was triple the national average—but by 2016, violence had nearly doubled, with over 780 killings and 4,500 shootings.

In 1990, Chicago and New York had had similar murder statistics. / Sources: Census, FBI, CPD

Crime has declined somewhat since. But the surge in violence put the spotlight on a disturbing long-term trend: most of Chicago’s murders (and almost all of its nonfatal shootings) go unsolved.

In 2016, just 5% of shootings resulted in an arrest, down from an already dismal 7-10% in prior years. So far this year, police have solved or “cleared” fewer than 7% of Chicago’s nonfatal shootings.

Source: UChicago

The homicide picture is nearly as bad, and it’s also part of a long-term trend.

In the 1960s, Chicago police cleared over 90% of homicide cases. In 2017, CPD's same-year murder clearance rate hit a new low: 17.5%.

Sources: WBEZ-Chicago

Police aren’t doing an amazing job of solving old cases, either. According to a crime database maintained by CPD, since 2001, more than half of of Chicago's murders are unsolved—involving over 4,900 victims—have not resulted in an arrest to date. And the longer it takes, the less likely it is that a case will ever be solved.

Unsolved Murders: A National Crisis

Chicago is not alone. Clearance rates in other big cities wracked by recent crime waves, including Baltimore, New Orleans, and Detroit, are also in long-term decline.

Sources: MAP, FBI, CPD

Nor is it only large, high-crime cities. Murder clearance rates have been falling nationwide for half a century, according to FBI data compiled by the Murder Accountability Project.

In 1965, the nationwide clearance rate for homicide was 91%; in 2017, it was just over 61%.

Source: MAP, FBI

What Percentage of Murders Are Solved? Clearance vs. Justice

When you try to follow murder cases after the police declare them solved, the problem just gets bigger.

First, many murders aren’t cleared by arresting a suspect. Instead, they are cleared administratively or through “exceptional means,” when the police decide (to their satisfaction) that they’ve solved the case but for whatever reason can’t arrest the suspect (for instance, if the prosecutor declines to issue an arrest warrant).

Starting in the mid-1970s, police in Chicago started clearing a lot of cases administratively: between 1975-1995, nearly 10% of the city’s murder clearances came by exceptional means.

Data compiled by the Washington Post show that exceptional clearances have become relatively more important as arrest rates have continued to slide. Between 2007 and 2017, over 21% of Chicago's increasingly rare clearances came by exceptional means; the average for 55 major cities was about 10%.

When New Murders Close Old Cases

Some cases are closed because the prime suspect is dead—for instance, if they commit suicide, are shot by police, or get murdered themselves.

In Baltimore, clearing old murder cases by pinning them on new murder victims is known as putting “bodies on bodies,” and Baltimore police have cleared dozens of cases this way in recent years. According to the Baltimore Sun, BPD has managed to boost its murder clearance rate from 30% in 2015 to over 50% in 2017, in large part by doubling the number of cases closed administratively.

While some murder victims were undoubtedly involved in previous crimes, closing a case this way can leave lingering doubts. There are no legal or national standards for clearing a murder this way, and deceased suspects cannot defend themselves. Often, the victim’s family is not even notified of the accusation.

Other exceptional closures are labeled “unable to prosecute.” Generally, that means the prosecutor doesn’t think there’s enough evidence to get a conviction, but the police say they’ve done all they can. Again, that may often be true, but it underscores the fact that “case closed” doesn’t always mean justice was served.

Justice: Neither Swift, Nor Sure

Even when police do make an arrest, that doesn’t necessarily mean they got the right guy—or that the case can hold up in court. There is no national system for tracking prosecutions after police make an arrest, but the numbers we have are not reassuring.

In 2009, the Bureau of Justice Statistics (BJS) conducted a survey of the 75 largest counties in the United States, which are collectively responsible for about half of all murders. One year after an arrest, only a third of murder charges had any kind of resolution (whether a conviction or acquittal); after two years, fewer than two-thirds were resolved.

The BJS called it quits after two years, so we don’t know what happens when it takes longer than that to get a decision. But of the cases that did reach an outcome, only 60% resulted in a conviction for murder, while 10% were convicted of some other charge (which may or may not have been related to the killing).

Fully 30% of cases were dismissed or acquitted outright.

Source: BJS

All this assumes for the sake of argument that people convicted of murder are always guilty, which they are not—for instance, a study of death penalty cases estimated a wrongful conviction rate of 4.1%.

The bottom line: With clearance rates around 60% and conviction rates around 60-70%, the evidence suggests that most murderers are getting away with it, and most victims’ families will never see justice done. In cities like Chicago, Baltimore, and Detroit, punishment for murder is actually the exception, not the rule.

How We Got Here

The growing trend of unsolved murder spans half a century, and it has left hundreds of thousands of cold cases in its wake. According to Thomas Hargrove, founder of the Murder Accountability Project, there have been over 220,000 unsolved murders in the United States since the 1960s.

This is not the world according to CSI: Miami and Law & Order, where cutting-edge technology always finds the killer and gravelly-voiced prosecutors almost always get a conviction.

TV is fantasy, of course, but it still seems impossible that it’s harder to solve murders today than at any time in the past. We have DNA evidence, cell phone tracking, license plate scanners, and dozens of other new tools. The FBI has collected fingerprints from over 100 million people, facial recognition data for 117 million, and DNA profiles on over 18 million.

The privacy costs of these systems are not trivial, but when it comes to solving the most serious crime, the benefits aren’t showing up in the bottom line.

This is arguably the biggest mystery in American criminal history. What’s going on here?

Bad Data?

One prime suspect in any statistical puzzle is bad data. If our data isn’t good, maybe there’s no mystery at all. In fact, figures collected by the FBI and published by the Murder Accountability Project show that police often fail to report information on thousands of murders.

In particular, during the heyday of high clearance rates in the 1960s, the share of homicides actually reported to the FBI was lower than in later decades. But even if we assume that every unreported murder goes unsolved, clearance rates would still have declined substantially, from around 75% in the mid-1970s to about 55% today.

Sources: MAP, FBI, author's calculations

The under-reporting theory also fails to explain why the trends for so many individual cities (which are easier to verify than the aggregate national figures) so closely mirror the nationwide trend. It also doesn't account for the huge variance between cities today, some of which still have murder clearance rates near the highs of decades past. (More on that below.)

The Miranda Effect?

In the 1990s, some scholars began pointing the finger at Ernesto Miranda—or at least the landmark Supreme Court case that bears his name. The 1966 decision in Miranda v. Arizona required police to issue so-called Miranda warnings to suspects taken into custody, advising them of their right to remain silent and to have an attorney present during questioning.

Some argued that this made police work more difficult by discouraging confessions. That sounds plausible, but the evidence for it is thin. The biggest problem is that the trend doesn’t line up. In fact, overall crime clearance rates had been dropping since at least 1959, well before the Miranda case—and subsequent cases that reduced Miranda rights did not cause any notable uptick in solving murder.

Moreover, the “Miranda Effect” theory (and others that blame protections for criminal suspects granted in the 1960s) cannot explain the huge variation between cities today. Miranda warnings are the law of the land across the country, but data collected by the Washington Post from 55 major US cities show that murder clearance rates vary enormously.

In any given year, the gap between the best and worst cities' clearance rates was a full 50 percentage points.

Source: WaPo

Another clue is that closure rates are not random: year after year, some cities consistently solve a high percentage of murders, while others don't. Moreover, the departments with the lowest clearance rates, such as Baltimore and Chicago, are not renowned for their defense of suspects’ civil rights, so there’s no obvious connection there.

Source: WaPo

The War on Drugs?

Other researchers suggest that the drug war has made American police much worse at fighting crime. They argue that decades of harsh anti-drug tactics have made many citizens (especially those in poor, urban, and black neighborhoods) less willing to call 911 or cooperate with the police. Commentators have pointed to stories about shocking police shootings, unconstitutional stop-and-frisk programs, and police corruption that may have shaken the public’s faith in law enforcement.

Statistics also show that murders connected to gang or drug disputes are less likely to be solved than other killings (which makes sense, given that everyone involved is part of an ongoing criminal operation). At the same time, the diversion of police resources toward drug offenses must, in some sense, reduce what’s available for dealing with violent crime.

Chicago, in particular, decided on a very risky anti-gang strategy, sometimes known as “decapitation.” CPD went after the city's organized crime hierarchies, taking out the leadership structures of the drug gangs. That plan basically worked, and the city’s biggest gangs splintered into smaller, less organized groups.

Unfortunately, this created vastly more opportunities for conflict, but it also meant that fewer people were “in the know” when someone was killed—and that, in turn, meant fewer people who could potentially cut a deal by turning in a murderer.

But there’s a limit to how much even this can explain. Murder clearance rates have fallen steadily, decade after decade, regardless of the intensity of the drug war, gang violence, or changes in “tough on crime” tactics.

The Cycle

A final possibility is that low clearance rates themselves drive low clearance rates. As Thomas Alexander, a University of Maryland professor, told the Chicago Tribune, “It's reflective of witnesses that don't want to come forward; they're afraid gang members will retaliate.”

According to one report, half of all gunshot victims in Chicago refuse to talk to the cops about it. Police say that’s the result of a “don’t snitch” code, but with thousands of shootings and an arrest rate of just 5%, “snitching” looks like a low probability gamble.

And even when people do come forward, they don't always get a timely investigation: it took Chicago police more than two weeks to look at the case of one of the 75 shooting victims from that bloody weekend in August.

If people don’t trust that police will actually put the criminals behind bars—or keep witnesses safe—they won’t come forward to testify. That fear of retaliation makes it harder to solve cases, which increases the fear of retaliation, and so on, driving the city into a high-crime, low-clearance equilibrium.

However it got started, this is the kind of machine that can drive itself.

What We Can Do About It

The unsatisfying truth is that we don’t have a simple answer to what happened to America’s clearance rates, and we don’t know exactly how we got to this point. But the good news is that we have some clues about how we might get out of it.

While many studies have found that luck plays a big role in solving murders—such as when fingerprints happen to be found on a door, or a random bystander identifies the suspect—police efforts really can make a difference.

Good Detective Work Matters

In Boston, a recent study found that devoting more time, officers, and resources to murder investigations made a significant difference in outcomes.

To try to boost its low homicide clearance rates, Boston PD assigned more detectives and evidence technicians to each case, collected and analyzed more evidence (such as video, fingerprints, and DNA), and interviewed more witnesses.

That extra effort paid off. The probability that at least one search warrant would be issued increased from 60% to 70%, and the probability that the murder would be solved went up from 47% to 57%. Controlling for other factors, the study concluded, “the BPD homicide intervention was associated with a statistically-significant 43.4% increase in the odds that a homicide case was cleared.”

Clearance rates even went up for typically hard to solve cases, including those involving guns, drugs, gangs, outdoor crime scenes, and young black or Hispanic victims:

The BPD homicide clearance intervention improved the odds that homicide cases with these challenging characteristics were cleared. For instance, the predicted clearance probability of a homicide case involving a 24-year-old black male killed outdoors with a firearm as a result of a gang-related dispute was (27.6%) during the pre-intervention time period and (43.1%) during the intervention time period, holding the other variables constant.The (15.5%) increase in the clearance odds suggests the BPD intervention generated a noteworthy improvement in the odds that homicide offenders were held accountable in these difficult to solve cases.

Notably, this increase in clearance rates bucked the trend for nearby towns and comparable US cities over the same time period, suggesting that the effect wasn’t a coincidence.

Incentives Matter

This is not the only paper that suggests extra effort can make an impact. A study in 2006 found a connection between cops’ pay and their clearance rates. The paper found that when police in New Jersey won an arbitration dispute over their pay—if they got as much or more than they asked for—clearance rates went up, and when they lost the dispute, clearance rates went down.

The probability that an arrest resulted in a conviction was also connected to the outcome of these pay disputes, suggesting that the quality of police work can change based on employee morale. The impact for murder was smaller than for other, more common crimes, like robbery and auto theft, but, in principle, the study showed that police effectiveness can be increased or decreased by outside forces.

A fascinating paper published this summer discovered another surprising fact: when local governments rely more on fines collected by cops, police solve fewer crimes. Controlling for other factors, a 1% increase in the share of city revenue from fines and fees was associated with a decrease of 3.7 percentage points in the violent crime clearance rate.

The researchers concluded that when towns get a larger share of their revenue from issuing tickets, collecting fees, and seizing assets (like cash, cars, and houses), cops spend more time extracting money from residents and less time solving serious crimes. This was especially true in towns with smaller departments, where officers are less specialized and can easily switch back and forth between different jobs—again suggesting that attention and effort make a difference.

Conclusion

Ultimately, these studies could suggest some specific reforms, like hiring more detectives or reducing towns’ reliance on tickets and fines, but the biggest takeaway might simply be that it is possible to learn things about how and why police solve crimes, if we’re willing to look. And, if we’re willing to try, it’s also possible to change the system to do that.

Some cities are much better at solving murder than others, and we should be striving to identify lessons from the places that do it well. More resources, better incentives, closer community relations, and even new technology (like computer algorithms to detect patterns in crime) could all play a role in reversing America’s declining clearance rates.

We don’t know exactly what the formula is, and it will probably be different in every city, depending on the type of crime it has. We didn’t get into this situation because of a single mistake, and we won’t get out of it because of a single fix. Instead, we’ll need lots of new studies, new experiments, and new ideas until we find out what works.

The evidence shows change is possible, but it will take thinking differently to make a difference.

This article has been updated to include figures on under-reporting.