We should not let Benedict XVI go quietly. One hesitates to say so, because he is elderly and frail and, much more importantly, because he is revered by many millions. Outsiders should tread warily, mindful that the papacy is central to Catholics' faith, even to their very identity. We ought to signal from the start that we mean no attack on Catholics or their beliefs when we say that the departing occupant of that high office has a moral, if not legal, case to answer. But such a case there is.

The heart of the matter is the rape and abuse of children by Catholic priests. The child abuse scandal in the Catholic church has spread to some 65 countries, with victims estimated to be in the many thousands: one survivors' group has 12,000 members, each with a heartbreaking story to tell. There will be many more victims who have stayed silent. Few would deny that this is the greatest single moral issue confronting the church.

For some, Benedict has proven himself on the right side of this most searching question. They note that he has closed loopholes in canon law, that he has centralised the handling of cases – rather than allowing each diocese to do its own thing – and that he has, above all, apologised on behalf of the church. In 2010, as cases emerged with alarming frequency – not just in the US, where the first major revelations came to light, but in Germany, Switzerland, Holland and elsewhere – the pope sent a message to the Irish victims of abuse: "You have suffered grievously and I am truly sorry." He acknowledged that their dignity had been "violated" and said the guilty men would "answer before God".

His defenders further note the action he took in the specific case of Marcial Maciel, the Mexican priest who had abused seminary students for decades and had secretly fathered sons – boys Maciel had also raped and abused. Early in his papacy, Benedict stripped Maciel of his ministry and confined him to a life of "penitence and prayer". To the new pope's admirers, such resolve was the natural consequence of his previous job, heading since 1981 the Congregation for the Doctrine of the Faith – the office previously known as the Inquisition. Shortly before becoming pope, the then Joseph Ratzinger was tasked with reviewing some 3,000 outstanding abuse cases. It's said that he read the harrowing details and "became disgusted".

But that account is, sadly, far from the whole picture. For Benedict has to be judged not only by his record as pontiff, but also in that pivotal, earlier role as Vatican enforcer. Among his responsibilities over many years was deciding the fate of those priests accused of crimes, including child abuse. It means that if the charge against the Vatican is that it turned an institutional blind eye to children's pain across several decades – hushing up the evidence, covering for rapist priests, moving them from diocese to diocese – then the blindness was, in significant part, Benedict's.

The evidence of church delay and indifference, if not obstruction, throughout the 80s and 90s is copious – and it came about when the now departing pope was the Vatican's most senior official, second in this matter only to John Paul II. So rather than giving Benedict credit for dealing with Maciel in 2006, we should be asking why Ratzinger did not deal with him much, much earlier.

But the guilt is not only of this institutional variety. Those decisions Ratzinger took directly are equally suspect. When he was the archbishop of Munich in 1980, the case of Peter Hullermann crossed his desk. Father Hullermann was accused of multiple crimes of abuse. In one case he had taken an 11-year-old boy hiking in the mountains, plied him with drink, locked the door, stripped him and forced him to perform oral sex. Yet Hullermann's punishment was simply to be moved from Essen to Munich for therapy. Within days, this known sexual predator was given pastoral duties with access to young people – and he promptly abused again. Benedict's defenders have long insisted those fateful decisions were taken by his deputy. But the crucial documents, when they surfaced, said otherwise.

No less disturbing is the case of the California priest Stephen Kiesle, convicted of tying up and molesting two young boys in a church rectory. His superiors wrote to Rome in 1981, requesting that the abuser be defrocked, warning of "scandal" if he remained. After an initial request for more information, Ratzinger took four years to deliver his reply. It came in Latin – and said his office needed more time to consider the case. No doubt grateful for the delay, Kiesle was able to return to one of his former parishes – in the youth ministry.

It was a similar story with Father Lawrence C Murphy of Wisconsin, tormentor of as many as 200 boys in his care at a special school for the deaf, telling them God wanted him to teach them about sex. Eventually the archbishop of Milwaukee wrote to Rome – to Ratzinger – demanding action. Once again, the future pope failed to answer. Eventually a secret, canonical trial of Murphy began in 1996, ordered by Ratzinger's deputy. But the trial was halted after the abuser wrote a personal plea to Ratzinger, requesting that he be allowed to "live out the time that I have left in the dignity of my priesthood". He was granted his wish, dying peacefully, buried in his priestly vestments. Those children, deaf and especially vulnerable, never saw justice.

Whatever warm words he uttered as pope, it is this record of action – and inaction – that matters more. Benedict never acted against the top echelon of cardinals and bishops who had covered up crimes and obstructed justice. After Cardinal Law had fled Boston – just before state troopers arrived bearing subpoenas over claims of child abuse by priests in his archdiocese – he found safe harbour in Benedict's Vatican. When those anxious to prosecute their abusers asked that the Vatican archives be opened, he kept the files shut. The truth is, what little this pope did to deal with the evil of child abuse came too late, and only under duress.

As an ex-pope, Benedict will no longer enjoy the sovereign immunity available to heads of state. A prosecution is possible – but only in theory. Who would hand him over? Certainly not the Italian authorities. For all that, despite his age and the reverence of the office he soon vacates, he should answer for his actions. Not only in the next life, but here and now.

Twitter: @j_freedland