Pope Benedict XVI was always going to be on a hiding to nothing. Following the charismatic and long-reigning John Paul II, Joseph Ratzinger, the shy, elderly German academic, fond of cats and playing the piano, who had spent decades burrowing in the Vatican bureaucracy, was unlikely ever to set the Tiber on fire, let alone thrill the planet's 1.2 billion Roman Catholics – a sixth of the world's population.

Long papacies are usually followed by short ones – the church gets tired of supreme leaders who occupy the throne of Saint Peter for too long. But those who are in power briefly can also have a marked effect on the institution – as the 20th century's other great pope, John XXIII, demonstrated in just five years. In Benedict's case, however, there has been a sense of drift and disappointment. "I'd say it has been disastrous," said Michael Walsh, the British historian of the papacy, on Monday night.

The problems facing the church remain: the child abuse scandal has not been resolved, nor has the church's loss of authority and self-confidence been reversed. In the west, and Europe was clearly the focus of Benedict's interest, the decline in church attendances and the lack of vocations to staff the future priesthood, the sheer disintegration in its status and esteem, have been neither confronted nor resolved. Catherine Pepinster, editor of the Catholic weekly the Tablet, said: "It has been a very troubled time. We have not got a Catholic church at ease with itself."

Indeed, the Vatican has seemed to be pressing hard in the opposite direction: into a cul-de-sac of conservative authoritarianism which neither inspires nor revives the mass of cradle Catholics, who are still deserting the church even in heartlands such as Spain and Ireland. Fifty years ago, governments in Catholic countries would tremble at the Vatican's displeasure; now they just wag their fingers back and press on with their plans for gay marriages or easier abortion. There is no comeback when the church has squandered its moral authority across the world over child abuse.

With the best will in the world, the push to revive the Latin mass, the imposition without much consultation of a ponderous new liturgy in English that looks as if it has been stitched together by someone whose first language is gobbledegook, and the serious attempt to entice the posturing reactionaries of the breakaway Pius X Society – including the Holocaust-denying English "bishop" Richard Williamson – back into the fold have been abject in purpose and effect. The church does not really do consultation at the best of times, but some of this has verged on the provocative: two fingers up to the laity.

At the same time, the creation of the Anglican ordinariate – a refuge for vicars finally unable to stomach the idea of female bishops in the Church of England – was not only undertaken negligently, without meaningful consultation, but also without much sense or reason. Those who thought the old authoritarian Vatican was coming back, but without the means or moral authority to make much difference to the fate of the church and its people, have been proved all too correct. This has been all the more unfortunate because Benedict got on well with Rowan Williams, another shy, bookish academic who himself surprised his flock last spring by deciding to retire early as archbishop of Canterbury. One of the few high points of his papacy was a successful visit to Britain in 2010.

On Monday there was already speculation about why Benedict should choose to be the first pope since the middle ages to retire: was he – like Harold Wilson when he stepped down as prime minister in 1976 – aware of a decline in his previously acute mental powers? Or was he harder hit by the papal butler scandal of last autumn, when a servant he trusted was convicted of passing leaked documents? Benedict had suggested in an interview a few years ago that he might retire, but in an institution where that hasn't happened for hundreds of years, his decision certainly took world Catholicism by surprise.

There is a small glitch that he has thrown into the works: his predecessor, Pope John Paul II made provision for any deadlock in a papal election to be short-circuited in the conclave, with a 50% plus one majority after the fourth ballot to replace the two-thirds majority initially required. Benedict has reverted to the previous system, meaning the likely formation of rival blocking minorities and a prolonged conclave. At least he will be able to watch the confusion.

Benedict was always going to be an interim pope, but he has not been an old man in a hurry so much as confused in facing a world where the church's old certainties and expectations are under greater challenge than ever before. It is said that Pope John XXIII was once asked what depressed him most about being pontiff. "It is going to sleep every night in the bed you know you will die in," he answered. At least Benedict will be saved that.