It is a truth beyond cliché that those outside the world of prisons rarely hear or read about what happens in prisons – unless there has been a riot or a particularly violent death in custody.

We see news film of angry inmates on roofs hurling tiles onto watching police or prison officers; we see smoke spiralling up from a fire deliberately started by prisoners upset at some real or imagined grievance; we hear ministers, Conservative and Labour, saying that there will be an inquiry and that lessons will be learned. Then we move on to the next news story or whatever concerns us more and forget about prisons, prisoners, and prison officers, and allow the metaphorical tide to re-cover the sand.

Even when the chief inspector of prisons publishes an annual report, as Peter Clarke has just done, it gets scant attention in the media and still less in parliament. That is almost as great a scandal as the things Clarke writes about in his Annual Report 2016-17 published this July.

You do not have to read very far into the document to become depressed, both because it is true and because it could have been written each year for the last 30 years. Clarke begins:

At the heart of our work is the inspection of adult prisons, which hold more than 81,000 men and nearly 4,000 women. Last year I reported that too many of our prisons had become unacceptably violent and dangerous places. The situation had not improved – in fact, it has become worse. There have been startling increases in all types of violence. The biggest increase is assaults on staff which, in the 12 months to December 2016, rose by 38 per cent to 6,844 incidents. Of these 789 were serious, an increase of 26 per cent. In total there were more than 26,000 assaults, an increase of 27 per cent. Of the 29 local prisons and training prisons we inspected during the year, we judged 21 of them to be "poor" or "not sufficiently good" in the area of safety.

This is pretty stark stuff. It is evidence of repeated failure on a large scale. It is evidence that should make us ashamed. But it is only part of the picture. Here is Clarke again:

By February this year we had reached the conclusion that there was not a single establishment that we inspected in England and Wales in which it was safe to hold children and young people… there seems to be something of a vicious circle. Violence leads to a restrictive regime and security measures which in turn frustrate those being held there. We have seen regimes where boys take every meal alone in their cell, where they are locked up for excessive amounts of time, where they do not get enough exercise, education or training, and where there do not appear to be any credible plans to break the cycle of violence.

Things are getting worse, not better. They have been for some while. Yet governments, past and present, seem to be unable or unwilling to do anything of a practical nature to repair or reform the system.

The analysis of the facts of prison life and the work done in them by governors, officers, teachers, psychiatrists, drug teams, medics and other professionals produces the same recommendations I was learning about and researching over 10 years ago when shadow prisons minister.

Far too many young men in prison were addicted to noxious drugs or alcohol (it was and is not difficult to get hold of these substances). Far too many of them were mentally unwell (and not just depressed because they were in custody, a long way from home, frightened and isolated, although they were). Far too many of them were ill-nourished and physically unfit and took no or little exercise; far too many of them were functionally illiterate and innumerate; and far too many of them were leading lives devoid of purpose in prisons that were overcrowded and under-staffed. They were unsafe places to live and to work. They were holding stations for the more than 80,000 people the rest of us wanted to forget about.

As Clarke also points out, the treatment of and regime for those more than 3,000 prisoners on indeterminate sentences for public protection (IPPs) is far from satisfactory. Ten years ago, I was told by every prison governor whose prison housed IPP prisoners that they were the most difficult cohort to manage. They had no release date, they could not get onto the courses necessary to demonstrate they were no longer a danger to the public, and many of them did not understand the policy behind or the practical effect of their sentence.

In court, they had taken in only the number of months or years of their minimum tariff before they could be considered for release, and it had not dawned on them that they might still be there a decade or more later. In prison, years beyond their tariff, they became stuck in a hopeless spiral of decline, depression and surliness that would occasionally erupt into violence against staff or other inmates or self-harm.

This week the former chief inspector of prisons and current chief inspector of probation, Nick Hardwick, has published his report. It reinforces Clarke’s. IPP prisoners are still, some years after the sentence was abolished, being held in custody years after their tariffs and still there is a long backlog of cases awaiting consideration by the Probation Board. It is an appalling state of affairs and one that can only be dealt with if prisons policy gets further up the government’s agenda of priorities.

Theresa May was not directly concerned about prisons as home secretary but the Ministry of Justice needs both a prime minister and a secretary of state to take a close interest in prisons policy. David Lidington is a man of great intellect and humanity. He has not, I suspect, had much to do with prisons since his time as Douglas Hurd’s special adviser 30 years ago. There were only 45,000 people in prison then. He needs to bring a sense of purpose to this brief which has been bedevilled by too many justice secretaries since 2010, some of whom could, had they stayed longer in the job, perhaps achieved positive reform. And he needs a prisons minister who has the political clout that comes with seniority within the government, who has the active support of the secretary of state, and who has the interest and ear of the Prime Minister.

The heart of the problem in our prisons lies in overcrowding. The government knows that. What they do about it is the issue. Promising more prison staff is no longer good enough although necessary. The new recruits are, sadly, often not up to the job and anyway do not stay in the job long enough to be of value. There are more than 85,000 people in custody and not enough people to look after them. As a result prisoners spend far too much time in their cells doing nothing. If they are sharing a cell (and many are sharing cells designed for single occupancy) they have no privacy even to the point of having to defecate in sight of their cellmate.

Read more: “I'm to blame”: Blunkett's indefinite prison sentences

Prisons are, as Clarke points out, unsafe. They are also unproductive. Those who leave them on the completion of their sentences are more likely to reoffend that not; they are more likely to be unemployable than employable. If they went in illiterate they are likely to be illiterate on release; if they went in with an addiction, they are likely to be addicted on release. If a company ran a factory making products where 50 per cent of them broke down within six months of production or never worked in the first place, there would be a shareholders’ revolt and the management would be sacked. Years of failure or absence of improvement would not be tolerated. Why though do we tolerate it in our prisons system?

The easy answer to overcrowding is to build more prisons, but we cannot afford that. The other answer is to send fewer people to prison. Drug addicts should be rehabilitated outside prison; the mentally ill should be treated and helped within the health system. Prisons should be reserved for the dangerous, the violent and the depraved. The government has more to do than just dealing with Brexit and it should deal with our prisons urgently.

The Rt Hon Sir Edward Garnier QC was Conservative MP for Harborough 1992-2017, shadow prisons minister 2005-09, shadow attorney general 1999-2001 and 2009-10, and HM solicitor general 2010-12