In the 15 years since I’ve been doing this job, my hopes that I would see a decent and humane prison system in this country in my lifetime have seldom risen above the level of “unlikely”. And a few times I have believed the state of jails in England and Wales simply could not get any worse, only to see it do so. The lowest point, until now perhaps, came during the reign of the book-banning justice secretary, Chris Grayling, from 2012-15. But even after his unlamented departure, there has been little for reformers to cheer, with ever-rising tides of self-harm, violence, overcrowding and squalor. And I can barely recall the last positive prison inspection report I have read.

How a gardening scheme is reaping rewards for prisoners’ mental health | Clare Horton Read more

There have been brief moments of hope. The last justice secretary, David Gauke, seemed to have a grasp of the basics when he proposed the abolition of short sentences – six months or less. All experienced prison watchers know that prisoners serving such short stretches seldom leave the confines of the local jails the courts send themto, which have the worst conditions in the penal estate. He intended to replace short sentences with community service orders and substance misuse programmes, both of which have significantly lower reoffending rates than short spells in prison.

Before losing his cabinet place, Gauke urged the new prime minister not to take the traditional Conservative hard line on crime, but to “follow the evidence, of what works and what doesn’t”. He quoted figures from the Ministry of Justice showing that reoffending costs the taxpayer about £15bn a year.

Some hope. Boris Johnson has not only scrapped Gauke’s forward-thinking plans on short sentences, he has announced he will build 10,000 new prison places and end the “early release” of prisoners. The plans have actually been in place since 2015, and “early release” – at the halfway point of fixed-term sentences – is dependent on the prisoner’s good behaviour. But why let the evidence stop the flow of rabble-rousing rhetoric? Let the public believe that we have a “cock-eyed, crook-coddling criminal justice system” in place.

Depressingly, this approach may work for Johnson in electoral terms. Prisoners can’t vote and don’t garner much sympathy from those who can. Yet we already lock up more of our citizens, and for longer, than any other country in western Europe – with appalling results.

Can you imagine the public sustaining a health service where the majority of patients don’t get better? So why continue to pour money into a penal system that clearly fails both prisoners and public?

When I first entered prison as an adult, in the 1960s, the No 1 rule in the prison service manual read, according to my memory, the purpose of imprisonment was “to teach prisoners to lead a good and useful life”. A pretty sensible commandment. Half a century on, prison rehabilitation projects have slipped further and further down the prison pecking order, with containment and punishment the priorities.

A fortnight ago, Johnson spoke to the media from outside HMP Leeds, a jail that encapsulates all the problems found in many local prisons. It is old, overcrowded and unsafe. The last inspection report in 2017 found conditions had deteriorated since the previous inspection two years earlier – which saw the jail failing three of the four healthy prison tests. This time, inspectors reported, amid a host of failings, “several staff had been suspended or dismissed for misbehaviour when using force”. Inspectors noted the “particularly troubling fact that, since the last inspection, there had been four self-inflicted deaths at the jail and another apparently similar death during this inspection”. They record: “The day after the inspection ended, there was an apparent homicide, and a few days later, another apparently self-inflicted death.”

A question to the prime minister: are the conditions you saw at Leeds, but failed to mention, bad enough to make criminals “afraid”, as your home secretary wishes? Or do you propose to make them worse?

The Guardian view on Boris Johnson’s prison policy: a fantasy built on an invention | Editorial Read more

Gauke’s successor at the justice ministry, Robert Buckland, has kept a relatively low profile since entering parliament in 2010. But the QC’s voting record, on serious social issues, does not point towards him adopting some of his predecessor’s forward-thinking ideas, given he has consistently voted against same-sex marriage, gay rights and increases in state benefits. The one hope is that his legal background will encourage him to follow the evidence on prison reform.

• Eric Allison is the Guardian’s prison correspondent