One of my most treasured possessions is The Art of Computer Programming by Donald Knuth, a computer scientist for whom the word “legendary” might have been coined. In a way, one could think of his magnum opus as an attempt to do for computer science what Russell’s and Whitehead’s Principia Mathematica did for mathematics – ie to get back to the basics of the field and check out its foundational elements.

In computer science, one of those foundational elements is the algorithm – a self-contained, step-by-step set of operations to be performed, usually by a computer. Algorithms are a bit like the recipes we use in cooking, but they need to be much more precise because they have to be implemented by stupid, literal-thinking devices called computers.

Algorithms are the building blocks of all computer programs and Knuth’s masterpiece is devoted to their analysis. Are they finite (ie terminating after a finite number of steps)? Is each step precisely defined? What are its input and output? And is the algorithm effective? In the broad sweep of his magisterial inquiry, however, there is one question that Knuth never asks of an algorithm: what are its ethical implications?

One in three black men can expect to be incarcerated (compared with one in six Latinos and one in 17 whites)

That’s not a criticism, by the way. Such questions weren’t relevant to his project, which was to get computer science on to a solid foundation. Besides, he was writing in the 1960s when the idea that computers might have profound social, economic and political impacts was not on anybody’s radar. The thought that we would one day live in an “information society” that was comprehensively dependent on computers would have seemed fanciful to most people.

But that society has come to pass, and suddenly the algorithms that are the building blocks of this world have taken on a new significance because they have begun to acquire power over our everyday lives. They determine whether we can get a bank loan or a mortgage, and on what terms, for example; whether our names go on no-fly lists; and whether the local cops regard one as a potential criminal or not.

To take just one example, judges, police forces and parole officers across the US are now using a computer program to decide whether a criminal defendant is likely to reoffend or not. The basic idea is that an algorithm is likely to be more “objective” and consistent than the more subjective judgment of human officials. The algorithm in question is called Compas (Correctional Offender Management Profiling for Alternative Sanctions). When defendants are booked into jail, they respond to a Compas questionnaire and their answers are fed into the software to generate predictions of “risk of recidivism” and “risk of violent recidivism”.

It turns out that the algorithm is fairly good at predicting recidivism and less good at predicting the violent variety. So far, so good. But guess what? The algorithm is not colour blind. Black defendants who did not reoffend over a two-year period were nearly twice as likely to be misclassified as higher risk compared with their white counterparts; white defendants who reoffended within the next two years had been mistakenly labelled low risk almost twice as often as black reoffenders.

Donald Knuth (left), who tackled algorithms in the 1960s in The Art of Computer Programming. Photograph: Bettmann Archive

We know this only because the ProPublica website undertook a remarkable piece of investigative reporting. Via a freedom of information request, the journalists obtained the Compas scores of nearly 12,000 offenders in Florida and then built a profile of each individual’s criminal history both before and after they were scored. The results of the analysis are pretty clear. If you’re black, the chances of being judged a potential reoffender are significantly higher than if you’re white. And yet those algorithmic predictions are not borne out by evidence.

A cynic might say that this is no surprise: racism runs through the US justice system like the message in a stick of rock. One in three black men can expect to be incarcerated in his lifetime (compared with one in six Latinos and one in 17 whites). That should be an argument for doing assessments and predictions using an algorithm rather than officials who may be prejudiced. And yet this analysis of the Compas system suggests that even the machine has a racial bias.

The big puzzle is how the bias creeps into the algorithm. We might be able to understand how if we could examine it. But most of these algorithms are proprietary and secret, so they are effectively “black boxes” – virtual machines whose workings are opaque. Yet the software inside them was written by human beings, most of whom were probably unaware that their work now has an important moral dimension. Perhaps Professor Knuth’s next book should be The Ethics of Computer Programming.