I serve on the independent monitoring board at Bedford jail where prisoners rioted this month. For the past eight years I have visited the prison about once a week. Every prison in England and Wales has an IMB of ordinary citizens, appointed by the justice secretary in much the same way as are magistrates. Their voluntary job is to monitor whether prisons do what they say on the tin – provide a safe and secure environment within which to help prisoners lead law-abiding and useful lives in custody and after they are released. Members visit their prison unaccompanied and unannounced, and have unfettered access to it.

In August, the IMB was so concerned for the safety of prisoners and staff that it took the unprecedented step of writing to the prisons minister, Sam Gyimah, asking for his urgent help.

HMP Bedford is a small, busy jail that holds about 500 prisoners, both remanded from the courts in neighbouring towns and others whose sentence plans are still being worked out prior to their dispersal elsewhere. It is located in the town centre, with excellent communications for staff and prisoners’ families, a location that allows, in principle, for extensive partnerships and engagements with local communities. But the antiquated buildings and facilities are desperately in need of renovation to make them fit-for-purpose (typically two grown men living, sleeping, eating and relieving themselves in cells built ungenerously in 1850 for one man, now with an unscreened WC and basin in place of two tin buckets).

The IMB’s last two annual reports – to the end of June 2015 and June 2016 – warned that the jail was in need of infrastructural investment, overcrowded by as much as 50%, and perilously under-staffed.

Within prisons drugs, violence and merciless bullying are endemic. The mix is toxic and threatening, a damaging punishment over and above the deprivation of liberty, autonomy and contact with your roots. Surely no judge intended such suffering when passing sentence?

Some lessons were learned a decade before I started visiting prisons: individuals need to be treated with decency, both because they are individual human beings and because rehabilitative efforts were shown to be futile in the absence of respect. Prisons can furthermore only be run economically by consent – an implicit pact between prisoners and staff, which depends upon the prisoners perceiving the prison management’s use of authority as having legitimacy. Decency thus has a crucial central place, in lock-step with rehabilitation, in securing cost-effectiveness.

The decency agenda has raised standards, but it requires continual attention which it has been often impossible to sustain in recent years as ”‘austerity” has swept through the prison estate. A series of savings-driven restructuring programmes have lowered salaries, changed terms of employment, drastically reduced staffing levels and deskilled officers. Capital and maintenance budgets have been slashed. Governors’ autonomy has been progressively eroded, not least by a long series of expensive and quite possibly ill-judged outsourcing contracts. The impacts have been devastating, and form the backdrop to the riots at HMP Bedford.

For prisons to deliver rehabilitation, on top of secure containment, there needs to be: a decent, structured, calm and uncrowded environment; sufficient numbers of committed staff who have the skill to go way beyond the (already very difficult) running of a safe jail, by working individually with every prisoner; ample opportunities for prisoners to use their time constructively (remedial and more ambitious education, training, work); and consistently superlative leadership and dedication to purpose throughout the system. By August this year, the cost-cutting programmes of the last few years had bitten hard into HMP Bedford. Many experienced staff members had left the service for better-respected and remunerated employment, necessitating the deployment of inexperienced officers. Recruitment and retention difficulties, (uncompetitive salaries set against the difficulty of the job) and long-term sickness were so serious that it was no longer possible to deliver the full range of required out-of-cell activities (opportunites to socialise with other prisoners, shower, make telephone calls, exercise, education, work etc) safely.

Opportunities for prisoners to be involved in any constructive employment were already drastically reduced and were fast vanishing. Instead, prisoners locked in their cells – often for up to 23 hours a day. These unplanned and unannounced “bang-ups” unsettled the prisoners as well as reinforcing their boredom. Staff shortages and rotations had become so severe that officers had insufficient time even to get to know the prisoners in their charge, let alone attend to their needs. The use of drugs had spiralled, and with that trade, the bullying of the weak and vulnerable. The incidence of self-harm and of violent assaults, both prisoner-on-prisoner and prisoner-on-staff, was rising alarmingly.

In effect he was saying: 'This is a prison! If you can't stand the heat, keep out of the kitchen'

There was increasing rumour of prisoner unrest. The IMB was hearing accounts from officers that they were becoming afraid to work on the prison’s largest wing. The IMB letter to the minister warned that staff shortages were “beyond crisis point” and of the “alarming rise in prisoners attempting to hang themselves”. In the 10 weeks between the posting of the letter and the minister’s response, two prisoners were found hanged.

Gyimah’s reply, dated 27 October, however, was written very much in the register adopted for the minister’s acknowledgement of the IMB’s annual report. It gave airy explanations of how the Ministry of Justice (MoJ) agency, the National Offender Management Service (Noms), and the senior prison officers were coping with the IMB’s concerns, while conveying nothing more urgent in intention than a pat on the head. In effect, he was saying: “This is a prison! If you can’t stand the heat, keep out of the kitchen. Sorry to hear that it has made you anxious.” Less than two weeks later, on Sunday 6 November, the prison erupted in a widely reported riot involving more than 200 men from two wings. The incident of “concerted indiscipline” stopped short of catastrophe because: the officers on duty read the seriousness of the situation in time to withdraw, by degrees, and deprive the rioters both of hostages and keys; because of the sang-froid of the governor, as she took charge of the incident in the “command suite”; and because the prisoners were intent more on making a protest than on blood. This last may be a surprise to many, but I believe it to have been the case. Two residential wings and the central office and observation area were comprehensively trashed, nonetheless.

Surveying the devastation a couple of days later, an IMB colleague and I were approached by two experienced officers, who were plainly deeply affected by the riot. They wanted us to see the graffiti – versions of “If you treat us like animals, we will eventually behave like animals”, which implied that, at that moment, the prisoners and officers saw the prison in the same light. These officers were acutely aware that the extreme working environment they had experienced for months was paralleled by the stress they had imposed on the desperate prisoners. Behind their emotion lay the cataclysmic loss of trust. When this collapses, then all that you have worked to achieve seems to have failed. The officers were devastated.

The prisoners involved in the riot have been sent, in small batches, to prisons across the country. When they return, as many surely will (they are essentially local men) after the wings have been refurbished and the staffing problems have been resolved, I expect to hear similar deeply felt stories of betrayed trust from them, because there is never only one side to a story.

An unverified video of the riot at Bedford prison, posted on social media on 7 November 2016. Photograph: PA

The inner turmoil experienced by the staff members on duty at the time of the riot has been debilitating. They had to take the awful decision to withdraw, abandoning the prison and vulnerable prisoners. The senior management team are racked with guilt that things came to this pass under their command.

Many rioters will only have participated for fear of violent reprisals if they abstained, creating fear at the time and complicated long-term consequences for them in the future. Even the instigators merit understanding – of their circumstances, if not of their actions. They may have been more hot-headed than others, but the prisoners had collectively been driven to desperation by the poverty and randomness of the regime. Nonetheless, prisoners and staff connived, in a sense, to bring the insurrection under control before any mortal damage was done. It is interesting that they left the library untouched.

It would be easy to attribute blame for the riot to the local actors, but that would represent a profound injustice and a failure to understand the horrors and complexities of prison. The staff at all levels at HMP Bedford have awed me by their consistent courage, loyalty, resilience, ready sense of decency and professionalism in the face of increasingly overwhelming odds. This was evidenced over months of difficulty and in spades on the day of the riot when prisoners, having stoically endured a seemingly endlessly deteriorating and irregular regime, lost faith in the prison management’s use of authority as having legitimacy, abandoned the implicit pact, and “kicked off”. They had cried for help, as had the IMB, the officers, and the governor to her superiors – but no one at the top of the tree heard their pleas. Who, then, was blameworthy?

I arrive at the following conclusions. The cost-cutting demanded of Noms, and, by extension, of individual prisons, has taken on the dimensions of an extreme human experiment, which in the social sciences these days would be blocked on ethical grounds. . There was no attempt to introduce these reforms gradually at a national level, or on a pilot basis, and then learn from the experience. There was no “evidence base” for these reforms. and no attempt to collect any. They were thus driven purely by ideology and finance. In general, the prison system seems reluctant to set up mechanisms whereby it can learn from its mistakes. The trial has failed also because the premise upon which it was based is misconceived. Prisons are being asked to achieve the unachievable.

Successive ministers have cut costs in the face of insistent warnings from many quarters, including the prison officers’ and the prison governors’ associations, the chief inspector of prisons, the IMBs and the many prison reform lobbying organisations. The only response has been about readjusting budgets over an extended period. Even though the new justice secretary, Liz Truss, has acknowledged (in her recent white paper) that cost-cutting has driven prisons to an extreme where they are not even able to operate safely, no proportionate or urgent action has been taken by the politicians and top civil servants in the MoJ who are responsible for the crisis in prisons. This is a devastating failure of accountability. They must acknowledge responsibility for the damage done.

Emergency services arrive at Bedford prison as the riot erupts. Photograph: Rex/Shutterstock

The recent prison riots are testimony to a slow-burn build-up of misery and conflict that has blighted the lives of the prison’s many communities. I would particularly highlight the intolerable burdens that have been placed on officers and managers, as this is seldom sufficiently recognised. At last, it is being acknowledged that prisons cannot be reformed without high-quality, motivated staff – and this is not simply about having the money to pay them a decent salary (although that is a necessary precondition).

Staff members have to be assured that their contribution is valued and that they will be trained and supported to do a difficult job in a meaningful way, especially when they are in the early stages of their career. Most prison officers do not go into the service to bang people up, but to help people change. Making the job so mechanical that it loses its meaning has led to a situation where it is as difficult to retain staff as it is to recruit them.

Criminal justice is a political playground, leading to endless changes in emphasis and policy. Prison budgets have not been ringfenced during “austerity”. In consequence, we have massively overcrowded and under-staffed older prisons, some of which are truly shocking, and that is unworthy of a nation that instinctively tends to assume it is a moral model for the rest of the world. Across the country, our prisons have failed to deliver against their social contract to “reform” offenders – so systematically that it is unclear that prisons (especially large ones) can ever deliver this while the political climate remains as it is.

Most prison officers do not go into the service to bang people up, but to help people change

A new vision is needed with an emphasis on integrating and accessing the insight contained within the system, . One such vision comprises three principal departures from current thinking. First, sentencing guidance would change radically to substitute community penalties for imprisonment for the vast majority of offences. for which research has shown that incarceration does not generally reduce reoffending. Second, there would be a properly resourced community sentence and probation service centre (CSPSC) in every major town to provide offenders with support, advice, help with housing and employment, remedial education and vocational training, alongside traditional probation service functions. Third, integrated into these centres would be small and local prisons so that all but the few prisoners who need high security jails more security and with intensive support to live safely, remain close to their roots. These new integrated centres would work most effectively if they formed a thriving and well-connected part of local life.

And what of the future for HMP Bedford? Its accessibility is such a huge asset that it merits the necessary investment, as long as the prison population is reduced to a level that allows it to offer personal attention, in a safe and calm environment overseen by an adequate number of experienced staff. Only then can a rehabilitative ethos have a chance to be effective.

It could even take part in the pilot roll-out of my proposed CSPSCs.

• Christopher Padfield is a member of IMB at HMP Bedford and former chair of the national Association of Members of Independent Monitoring Boards. The views expressed here are his own. To join the IMB for a prison near you go to imb.org.uk