If Beware were open source software, an unjust or flawed criteria or piece of code could be discovered. But the secrecy around Intrado’s approach makes real oversight impossible.

In that way, the company’s product is part of an alarming trend. More and more secret code is being incorporated into the criminal-justice system, making it more opaque and vulnerable to mistakes. “Proprietary software interferes with trust in a growing number of investigative and forensic devices, from DNA testing to facial recognition software to algorithms that tell police where to look for future crimes,” Rebecca Wexler explained last year in Slate. “Inspecting the software isn’t just good for defendants, though—disclosing code to defense experts helped the New Jersey Supreme Court confirm the scientific reliability of a breathalyzer.”

Dyer downplays the concerns of Beware's critics, assuring the Washington Post that its scores don’t trigger a particular response––in Fresno, he explained, 911 operators “use them as guides to delve more deeply into someone’s background, looking for information that might be relevant to an officer on scene.” But that just means that any faulty information would come to the police officers second hand. Dyer added that street officers never see the scores, although in Fresno, they were told about a “red” address and then called in a negotiator.

Police departments that use Beware would seem to have troublingly perverse incentives, insofar as cops who use deadly force are judged based on what a reasonable officer would’ve done in the same situation with the same information. Lawsuits against municipal governments turn on that question. Will the fact that police were responding to a call relating to a house or individual with a red threat level now be used to argue that subsequent force was relatively more reasonable?

I can imagine a “smart” emergency-response system that incorporates publicly available data in a way that enhances public safety without infringing on civil liberties. If I had a schizophrenic son, I would love a way to register that fact with the emergency-response system so that dispatchers would know to send folks trained to deal with the mentally ill. In the event of a fire, I’d love for my local fire department to know automatically that my wife and I have a friendly dog in the house.

But the number and gravity of the unanswered questions surrounding the technology that Fresno and others are using more than justifies the skepticism of critics like the ACLU of Northern California’s Matt Cagle, who summed up his objections to me:

Fresno police rushed forward with new surveillance technology without meaningful public input about whether this software is appropriate or what safeguards should be in place. This is yet another surveillance tool being used without transparency or accountability, and it risks targeting communities that are already vulnerable to police misconduct.

If Intrado opened up every aspect of Beware to scrutiny by the public, and the Fresno police force committed to specific protocols, its benefits might prove greater than its drawbacks. But so long as residents aren’t allowed to know what causes their local police force to stigmatize a person or an address as “dangerous,” they’re effectively denied the ability to decide whether the software tool is just or prudent. My threat assessment: Beware of this product and proceed only with great caution.

This article is part of our Next America: Criminal Justice project, which is supported by a grant from the John D. and Catherine T. MacArthur Foundation.