By Michele Catanzaro

This story is part of AlgorithmWatch’s upcoming report Automating Society 2020, to be published later this year. Subscribe to our newsletter to be alerted when the report is out.

This story is part of AlgorithmWatch’s upcoming report Automating Society 2020, to be published later this year. Subscribe to our newsletter to be alerted when the report is out.

As part of a program to curb feminicides, Spain built VioGén, an algorithm that assesses the risk faced by victims of gender violence. It remains a work in progress.

In the early morning of 24 February 2018, Itziar P., a psychologist living in the Spanish city of Castellón, went to a police station to report threats from her husband, Ricardo C..

In audio recordings she made with her mobile, the husband could be heard saying: “We will all end up dead and me in jail”, or “I will take away from you what you love most”.

According to Itziar P., Ricardo C. had also broken in pieces the buggy of their smaller child (Martina, 2 years old) and slapped the older one (Nerea, 6), when the two were under his custody.

The police officer asked Itziar P. a set of questions and fed the answers to VioGén, a software that helps the Spanish police estimate the risk of recidivism in gender violence. The officer issued a report in which that risk was deemed as low.

Critical failure

In the following days, both Itziar P. and Ricardo C. were called to declare in court. She asked that he be forbidden to visit their children, but the judge denied the request, based on the low risk estimation made by the police, among other reasons.

Seven months later, on 25 September 2018, Nerea and Martina where sleeping at Ricardo C.’s place. In the early morning, he killed them “with cruelty” and threw himself out of a window.

Itziar P.’s story was shocking. Why was the case deemed as low risk? VioGén had missed its goal of supporting police personnel in assessing the risk of new assaults, and so assigning the right level of protection. Since the software was first deployed in 2007, there has been a series of “low risk” cases that have ended in homicide of women or children.

Better than nothing

The program is by far the most complex of its sort in the world. It has reasonable performance indexes. Nobody believes that things would be better without it – except the far-right, which makes spurious claims that it helps women report innocent men.

But critics point out some flaws. Few police personnel are educated in gender-based violence. Some of the others blindly rely on the software’s outcome. Moreover, the program may have systematically underestimated risk. Some victims’ organizations believe that the possibility of a low risk score is nonsense. Reporting to the police is a high risk situation in itself, they say, because abusers perceive it as a challenge.

As of January 2020, 600,000 cases had gone through VioGén. About 61,000 of them were considered active, meaning they were being followed-up by the police (the system is designed to check periodically on women until they are considered safe).

Reporting an assault

When a woman goes to report an assault from an intimate partner, she triggers a process that takes at least a couple of hours. First, the police agent goes through an online form with her. The agent ticks each of the items of the VPR form (from the Spanish initials of “Police Risk Assessment”) as “present” or “non present”. There are 39 items in the latest published version of the form (the VPR 4.0 ). Agents can also rely on police databases, witnesses and materials proofs.

Questions explore the severity of previous assaults (for example, whether weapons were ever used); the features of the aggressor (jealous, bully, sexual abuser, unemployed, drug addict, etc.); the vulnerability of the victim (pregnant, foreign, economically dependent, etc.); and aggravating factors (like assaults by other men).

Answers are thrown automatically into a mathematical formula that computes a score, measuring the risk that the aggressor repeats violent actions. This quantitative approach is different to the one used in DAS-H, the British equivalent of VioGén. The latter is basically a paper check-list that helps agents to get an idea of the situation.

Keeping the score

In theory, Spanish agents can increase the score by hand, if they appreciate a higher risk. But a 2014 study found that they stuck to the automatic outcome in 95% of the cases.

The formula used in VioGén is a “simple algorithm,” according to Juan José López Ossorio, a psychologist who has been in charge of VioGén from its early stages, in a written statement to AlgorithmWatch. The algorithm gives more weight to items that empirical studies have shown to be more related with recidivism, Mr López Ossorio wrote. He declined to disclose the exact formula.

Once a case’s score is established, the agent decides on a packet of protection measures associated to that level of risk . For the lowest scores, agents will discreetly check on the woman from time to time. For the highest, the police will give the victim an alarm button, track the aggressor’s movements, or guard her house. The agent also sends the form and risk score to the prosecutors and judges that will see the woman’s case.

After the first report, the police meet again with the woman to fill in a second form, in order to assess whether the situation has worsened or improved. This happens periodically, more or less frequently depending on the risk level. Police stops following up only if judicial measures are not pursued and the risk level falls below medium.

VioGén is one of the outcomes of a pioneering law on gender-based violence that Spain approved in 2004, ten years before the Council of Europe adopted a common framework on the subject, the Istanbul Convention. Nowadays, the software is used by the main Spanish police forces (Policía Nacional and Guardia Civil) and by hundreds of local police forces (except in Catalonia and the Basque Country that have independent police bodies).

The best available system

VioGén is the best device available to protect women’s lives, according to Ángeles Carmona, president of the Domestic and Gender-Based Violence Observatory of the Spanish General Council of the Judiciary (CGPJ).

She recalls a case she saw in a court in Seville, of an aggressor that had a high-risk of recidivism, according to VioGén. A control wristband was applied to the man. One day, the police saw that the signal of the wristband was moving fast towards the victim’s home. They broke into it just in time to prevent him from suffocating her with a pillow.

It’s impossible to know how many lives have been saved thanks to VioGén, according to Antonio Pueyo, a professor of psychology at the Univeristy of Barcelona who has advised VioGén from the beginning.

However, a 2017 study by Mr López Ossorio and his team tried to measure how good the protocol was. They found that VioGén’s Area Under the Curve (AUC), a widely-used measure of performance for predictive models, stood between 0.658 and 0.8. An AUC of 0.5 is as good as a coin’s toss and an AUC of 1 means the model never fails. Cancer screening tests are considered good when their AUC is between 0.7 and 0.9. In other words, VioGén works.

“Comparing with what is around and within the existing limitations, VioGén is among the best things available”, says Juanjo Medina, a professor of quantitative criminology at the University of Manchester that has compared instruments of assessing the risk of intimate partner violence.

Spain is the only place where victims can be followed-up across different regions. Close to 30,000 police officers and other agents across the country had access to VioGén in 2018.

However, the cases that have slipped through the cracks of VioGén have raised concern. The latest one happened in February 2020, when a 36-year-old woman and mother of two had her throat cut by her former partner, who then threw her body in a container in the town of Moraira. The two had been registered in the VioGén system after the police reported him for attacking her, but the case had become inactive after a judge cleared him.

False negatives

In 2014, the newspaper El Mundo published a leaked document from the General Council of the Judiciary that showed that 14 out of 15 women who were killed that year, having reported their aggressor before, had low or non-specific risk (the classification used for any person reporting a threat to the police).

Some critics say that low risk should not even be an option. Reporting is a maximum risk moment for a woman, according to Carme Vidal Estruel, spokesperson of Tamaia, an association that helps victims in Barcelona. She says that the situation is akin to divorcing or becoming pregnant, both moments in which the aggressor realizes that he is loosing grip on the victim.

Another widespread criticism is that few agents among those who should validate the computer’s outcome receive enough training in gender issues. Some VioGén items are embarrassing, like those related to sexual violence, humiliation, or intimate messages on mobile phones.

Agents should ask circular questions (instead of blunt, direct questions) and avoid transmitting the feeling that the object of investigation is the woman, according to Chelo Álvarez, president of Alanna, an association of former victims in Valencia. Ms Carmona of the General Council of the Judiciary recalls a woman that reported her husband for robbing her car’s keys. She was so scared that she could not say anything else. The day after, the man killed her.

Few agents are aware of these nuances. In 2017, there was a total of 654 agents in all Spain belonging to the Women-Children Teams (EMUME) of the Guardia Civil. That is much less than one for every police station.

Ignored requirements

This is very different from what the 2004 law that created VioGén required. According to it, cases should be dealt with by an interdisciplinary team including psychologists, social workers, and forensic doctors.

This team should go into psychological aspects that the VioGén form does not cover. Moreover, it should carry out a forensic assessment directly on the aggressor. The current system is tantamount to evaluating how dangerous is a person without ever talking with him, critics points out. Several teams were created after the law was passed in 2004, but the process was cut sharply by the austerity following the 2008 financial crisis.

Mr Pueyo, the psychology professor, acknowledges some of the criticism, but believes that VioGén should be judged for its ability to predict new assaults, not homicides, because these events are very rare. The probability that a women be killed after reporting is about one in ten thousands, according to Mr López Ossorio.

However, the Istanbul Convention requires precisely to reduce the risk of death. And not only of the women, but also of their children. Overlooking the risk for children is another criticism VioGén faces.

The convention entered into force in Spain in 2014, but VioGén forms were not changed accordingly until Itziar P.’s case occurred in 2018, according to her lawyer.

VioGén 5.0

A new protocol was put in place in March 2019, the fifth big change VioGén has gone through since its first deployment in 2007. Now, the program identifies cases “of special relevance”, in which the danger is high, and cases “with minors at risk”.

This is done through the “dual evaluation procedure” of the new VPR form (VPR 5.0 -H), Mr López-Ossorio explains. Two calculations are carried out in parallel: one related to recidivism and a new one related to lethal assault.

Depending on the outcome of the latter (called “H-scale”), the risk score can be increased automatically. Moreover, the case can be signaled to the prosecutors and judges as being “of special relevance”.

Mr López-Ossorio declined to disclose how the H-scale was built, but wrote that it was based on a study his group carried out throughout four years, to find which factors were specifically related to cases that end up in homicides.

The new protocol seems to have triggered a major shift in the risk scores of VioGén. Passing from VPR 4.0 to VPR 5.0 -H, the number of extreme risk cases rose and those of high risk almost doubled, according to Mr López Ossorio.

As the president of Valencia’s former victims association Ms Álvarez puts it: “Things are improving, but they should go faster, because we are being killed”.