Get a daily rundown of the top stories on Urban Milwaukee

Paul Zilly had been convicted of stealing a push lawnmower and some tools in Barron County in northwestern Wisconsin back in 2013. “The prosecutor recommended a year in county jail and follow-up supervision that could help Zilly with ‘staying on the right path,'” as a story by Pro Publica recounts. “His lawyer agreed to a plea deal.”

But Judge James Babler also had before him the results of Zilly’s scores on the Correctional Offender Management Profiling for Alternative Sanctions, or COMPAS test. The test, sold by the for-profit company Northpointe, and used widely in Wisconsin, rates an offender’s chances of committing more crimes in the future. It had rated Zilly as a high risk for future violent crime and a medium risk for general recidivism. “When I look at the risk assessment,” Babler said in court, “it is about as bad as it could be,” the story reports.

As a result, Babler overturned the plea deal agreed on by the prosecution and defense and instead sentenced Zilly to two years in state prison and three years of supervision.

The use of tests like this, known as risk assessments, “are increasingly common in courtrooms across the nation” and “used to inform decisions about who can be set free at every stage of the criminal justice system, from assigning bond amounts — as is the case in Fort Lauderdale — to even more fundamental decisions about defendants’ freedom,” Pro Publica reports. And “Northpointe’s software is among the most widely used assessment tools in the country… In Arizona, Colorado, Delaware, Kentucky, Louisiana, Oklahoma, Virginia, Washington and Wisconsin, the results of such assessments are given to judges during criminal sentencing.”

The objective was to make these decisions more scientific and objective. But in 2014, “then U.S. Attorney General Eric Holder warned that the risk scores might be injecting bias into the courts” and “called for the U.S. Sentencing Commission to study their use,” Pro Publica notes. But no such study was undertaken.

And so Pro Publica did its own study: its reporters obtained the risk scores assigned to more than 7,000 people arrested in Broward County, Florida, in 2013 and 2014 “and checked to see how many were charged with new crimes over the next two years.”

The results showed the COMPAS test was “remarkably unreliable in forecasting violent crime: Only 20 percent of the people predicted to commit violent crimes actually went on to do so.”

The results also suggested the test might be racially biased: “The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants.” Meanwhile, “white defendants were mislabeled as low risk more often than black defendants.”

“Could this disparity be explained by defendants’ prior crimes or the type of crimes they were arrested for?” Pro Publica asked. “No. We ran a statistical test that isolated the effect of race from criminal history and recidivism, as well as from defendants’ age and gender. Black defendants were still 77 percent more likely to be pegged as at higher risk of committing a future violent crime and 45 percent more likely to be predicted to commit a future crime of any kind.”

Since that story was published, in May 2016, “some of the nation’s top researchers at Stanford University, Cornell University, Harvard University, Carnegie Mellon University, University of Chicago and Google” decided to study the test, as a follow up story published by Pro Publica last week reports:

“The scholars set out to address this question: Since blacks are re-arrested more often than whites, is it possible to create a formula that is equally predictive for all races without disparities in who suffers the harm of incorrect predictions?

“Working separately and using different methodologies, four groups of scholars all reached the same conclusion. It’s not…The researchers found that the formula, and others like it, have been written in a way that guarantees black defendants will be inaccurately identified as future criminals more often than their white counterparts.”

Northpointe, the company that sells the COMPAS tool, said it had no comment on the critiques. As for Wisconsin, “State corrections officials declined repeated requests to comment” for the first story published by Pro Publica, it noted.

In response to my request for comment, Tristan D. Cook, Communications Director for the Wisconsin Department of Corrections, asked me to email my questions and then sent this response:

“In 2010, DOC selected COMPAS as the primary tool for the Department of Corrections. One of the primary functions of criminogenic risk and need assessments is to provide a score that reflects the relative criminogenic risk of a specific offender relative to the overall population. Criminogenic risk and need assessments can be used in a variety of situations, including pre-sentence investigations, inmate classification, supervision intake and reclassification, and release planning. Counties can also elect to use COMPAS. ”

Cook had no response to my main question, whether the department was concerned about the potential bias of the COMPAS test. But his response suggests the test can be used at every stage of the criminal justice system. Which is exactly what Pro Publica reported:

“Wisconsin has been among the most eager and expansive users of Northpointe’s risk assessment tool in sentencing decisions…In 2012, the Wisconsin Department of Corrections launched the use of the software throughout the state. It is used at each step in the prison system, from sentencing to parole.

“In a 2012 presentation, corrections official Jared Hoy (a Policy Initiatives Advisor for Wisconsin’s Department of Corrections) described the system as a ‘giant correctional pinball machine’ in which correctional officers could use the scores at every ‘decision point.’

“…Some Wisconsin counties use other risk assessment tools at arrest to determine if a defendant is too risky for pretrial release. Once a defendant is convicted of a felony anywhere in the state, the Department of Corrections attaches Northpointe’s assessment to the confidential presentence report given to judges, according to Hoy’s presentation.”

It seems remarkable that Wisconsin officials would have no concern about racial bias in a test used so widely, all the more so in a state criticized as a leader in the incarceration of African American males. As the watershed study by the UW-Milwaukee Employment & Training Institute found, Wisconsin leads the nation by far in the percent of black males who are incarcerated: statewide, 12.8 percent of all African-American males are incarcerated, nearly double the national average (6.7 percent) and well ahead of 2nd place Oklahoma (9.7 percent).

“State DOC records show incarceration rates at epidemic levels for African American males in Milwaukee County,” the study notes. “Over half of African American men in their 30s and half of men in their early 40s have been incarcerated in state correctional facilities.”

This problem goes back decades and predates the COMPAS test. But the question is whether the adoption of this test will undercut reform efforts (such as the creation of drug courts in cities like Milwaukee and Madison that aim to reduce the level of incarceration for non-violent offenders). Is the use of the COMPAS test as a “giant correctional pinball machine” throughout the system a wise decision given the evidence of its racial bias?

In the case of Paul Zilly, the man sentenced to two years in prison for stealing the lawnmower and tools, after he was sent to prison, a public defender appealed the sentence and called on an interesting witness: Tim Brennan, the former professor of statistics at the University of Colorado who had created the COMPAS test and later sold it to the company that now owns it. “Brennan testified that he didn’t design his software to be used in sentencing,” Pro Publica reports.

“After Brennan’s testimony, Judge Babler reduced Zilly’s sentence, from two years in prison to 18 months. ‘Had I not had the COMPAS, I believe it would likely be that I would have given one year, six months,’ the judge said at an appeals hearing…

“Zilly said the score didn’t take into account all the changes he was making in his life — his conversion to Christianity, his struggle to quit using drugs and his efforts to be more available for his son. ‘Not that I’m innocent, but I just believe people do change.'”

Zilly, by the way, is white. Meanwhile, the cumulative impact of the COMPAS test on black offenders in Wisconsin is unmeasured and unknown.

If you think stories like this are important, become a member of Urban Milwaukee and help support real independent journalism. Plus you get some cool added benefits, all explained here.