Under an all-but-full moon on the opening evening of the Second Vatican Council in 1962, tens of thousands of Romans poured into St Peter's Square in a torchlight procession. Called to the window of his study by the multitude, John XXIII, the man Italians called "the good pope" (a term that speaks volumes about their view of the previous 260), delivered one of the great speeches of an eloquent decade. Its emotional high point came when he told the crowd: "Returning home, you'll find the children. Give your children a caress and say: 'This is the caress of the pope.'"

Forty-three years later, on the night Benedict XVI was elected, a text message hurtled between mobiles as the crowds dispersed from that same square. "Returning home, you'll find the children", it read. "Give them a belt round the ear and say: 'That's a belt round the ear from the pope.'"

In the five years since, it has become clear that the daunting reputation the former Cardinal Joseph Ratzinger brought into office as "God's rottweiler" was, in many respects, misleading. A man less likely to cuff a child would be hard to imagine – although, as the latest and biggest scandal to rock his papacy has revealed, his choirmaster brother was not above cuffing choristers.

On Saturday, Benedict tried to put the lid on that scandal with a letter to the Catholics of Ireland apologising for the "sinful and criminal" abuses of children whose disclosure has rocked the Irish church to its foundations and helped bring down a Dublin government. Using language rarely heard from popes, whose utterances on some issues are held by Catholics to be infallible, he said he was "truly sorry" for what had happened over a period of decades.

But, as at other crucial junctures in his papacy, Benedict's gesture left many critics – and admirers – feeling that he could have achieved more by being bolder. As the scandal over clerical paedophilia spread through Europe, more than one senior Vatican official suggested it would be diplomatic for the pope to extend his letter to include countries other than Ireland.

Words such as "diplomatic" and phrases such as "media impact", though, have little relevance for Benedict. He stuck to his original plan, with the result that Catholics in the Netherlands, Germany, Austria, Switzerland and Italy feel he does not care about similar outrages that have come to light in their own churches. Emotions in Germany are running particularly high about the silence of the first German pope for almost 1,000 years.

The affair highlights two characteristics of this most paradoxical of pontiffs: his timidity and his stubbornness. Benedict has a soft, plump handshake, and a tendency to cast his eyes downwards as you speak to him – a shyness that, associates say, goes hand in hand with genuine humility. He is also a man of temperate habits; though not teetotal, his favourite drink is said to be fresh orange juice.

Other than theology and philosophy, the pope's main known enthusiasms are for classical music (he is an accomplished pianist, with a special love of Mozart and Bach) and cats – and they also seem to have a fondness for him. Cardinal Tarcisio Bertone, his secretary of state, has recounted how, when Ratzinger was a senior Vatican official, "Every time he met a cat, he would talk to it, sometimes for a long time. The cat would follow him." On one occasion, Bertone said, Ratzinger had led an entourage of "about 10" cats into the Vatican and a Swiss Guard had protested that they were "invading the Holy See".

"In private, he is more informal than you might expect," says Marcello Pera, an Italian senator who collaborated with Benedict on a book published in 2004. "He loves to talk and cast off his [official] role. If you speak to Benedict, you soon cease to remember that you're speaking to a pope." However, a rueful subordinate describes him as "bloodless" – and, certainly, his fame as the late John Paul II's "enforcer" was not unearned.

Formidable self-belief

For 24 years before becoming pope, Benedict was head of the Congregation for the Doctrine of the Faith (CDF), the Vatican department (once known as the Inquisition) whose main task is to maintain doctrinal orthodoxy by ensuring that theologians licensed to teach in Catholic institutions keep within bounds set in Rome. Among the most famous cases he dealt with was that of the American theologian Charles Curran, who was suspected of straying from the Vatican's line on birth control.

In his biography of Benedict, John Allen recounts a chilling episode. On 8 March 1986, Curran held a meeting with the then Joseph Ratzinger and others in the CDF's offices in Piazza del Sant'Uffizio, next to St Peter's. Curran was arguing that his approach fell within the mainstream of contemporary theological thinking when "Ratzinger asked Curran to name others who held his views. Curran did so – pointedly, they were all German – and Ratzinger then asked if Curran would like to accuse these thinkers, because if so, the Congregation would be happy to open an investigation."

This hardline story begins to explain the apparent contradiction of the current papacy – that a man as mild as Benedict should have upset more people than perhaps any pope in history. His earliest and most notorious gaffe (if that is what it was) came in 2006. In a scholarly lecture at his old university at Regensburg, he quoted the words of the 14th-century Byzantine emperor Manuel II that all Muhammad had brought were "things only evil and inhuman". And that was just the start of his controversial musings.

He then dismayed Protestants by signing off on a document that relegated their churches to mere "ecclesial communities"; outraged indigenous representatives in Latin America by insisting that evangelisation in the New World did not represent "the imposition of a foreign culture"; and angered Jews by setting the controversial wartime pope, Pius XII, on the road to beatification. Last year, he appalled many Catholics – and not just on the progressive wing of his church – by agreeing to lift the excommunication of four breakaway, ultra-traditionalist bishops, one of whom turned out to be a Holocaust denier.

"I'm like the cellist Rostropovich," Benedict is said to have told a friend. "I never read the critics." Even by papal standards, his self-belief is formidable.

"He's shy – almost timid, I would say – but when he has to take a stand, he does it. When he thinks it's his duty, he can't be moved from saying what he feels," says Rocco Buttiglione. An Italian philosopher and politician who first met Benedict almost 40 years ago, Buttiglione knows all about the implications of voicing unpopular opinions: his own views on homosexuality cost him a place on the European Commission. And it is precisely Benedict's bluntness – especially with regard to Islam – that endears him to admirers such as Buttiglione. "There is so much ecumenism that is simply the exchange of meaningless words. Sooner or later we must get to grips with the real issues over Islam," he says.

However painfully, argues Buttiglione, Benedict's Regensburg lecture helped to do just that. "It provoked a real dialogue." Several initiatives have since been launched from the Muslim side, and two years ago, representatives of 138 scholars who signed a response – the Common Word declaration – met the pope at the Vatican.

Once regarded as a liberal

As the latest crisis has shown, though, Benedict's candour does not go hand-in-hand with a propensity for swift, decisive action. He might, for example, have won back a lot of ground for himself and the Vatican in Ireland by removing the Irish primate, Cardinal Sean Brady, after he admitted being involved in a shameful cover-up in the 1970s. Yet Brady remains in his job. The pope – who is to visit Britain later this year – may be fearless in some respects, but his personal history is one of growing fearfulness in others.

Unlikely as it may seem, Benedict was once regarded as a liberal, even a radical. As theological adviser to Cardinal Joseph Frings, the archbishop of Cologne and one of the decisive figures at the Second Vatican Council, a younger Joseph Ratzinger played a key role in awakening his church from centuries of reactionary slumber. Like many other young Catholic theologians of the time, he embraced the council's support for ecumenism, its enthusiasm for dialogue with other religions, its adoption of a liturgy in which worshippers could participate more actively, and its endorsement of greater internal democracy (in the form of increased autonomy for national Bishops' conferences). In 1966, he wrote a book that was translated into English as Theological Highlights of Vatican II.

"That book is absolutely brilliant," says a leading British liberal Catholic who, such is the atmosphere in today's Catholic church, asked not to be identified. "I've never seen the spirit of the council better exemplified. It's very moving. You feel that this is a man on fire." Intriguingly, Buttiglione uses precisely the same metaphor when describing his first encounter with Benedict, six years later: "He had an inner fire and great knowledge as well as a great capacity for listening."

By 1972, however, the flames kindling Ratzinger's intellect were different ones. In between, Europe had been shaken to the core by an eruption of student protest that began on the streets of Paris and raced across the continent, soon reaching the tranquil university town of Tübingen. Its revolutionary implications profoundly alarmed the then 41-year-old Ratzinger, the university's professor of dogmatic theology, and the following year he fled Tübingen for Regensburg, in his (more traditionalist) native region of Bavaria.

Buttiglione visited him there as part of a delegation from an emerging Catholic lay movement, Communion and Liberation, whose young members aimed to offer an alternative to the maverick student Marxism of the "68ers". Now, Ratzinger was one of their idols – having embarked on a process of intellectual change that would see him question many of the most important decisions reached by the Second Vatican Council.

Ratzinger and his supporters insist he has remained theologically consistent, but the Catholic author and academic John Cornwell disagrees. "He has not been a great champion of respect for other religions. He has not been a champion of giving a greater say to bishops, and nor has he supported the liturgy of participation – his own enthusiasm being for the Tridentine [pre-conciliar] Mass," which is normally said in Latin.

The man who once wrote so movingly about the council's embrace of the 20th century has become the pope who, as Cornwell notes, "gave a bishopric to an Austrian priest who preached that hurricane Katrina was a retribution for the sinfulness of New Orleans". But Benedict's supporters (and even some of his detractors) argue that to see him purely as a conservative is to do him an injustice.

"It would be interesting to know how he is viewed in a hundred years' time," muses a professional Vatican-watcher who also requested anonymity. "It will be either as a reactionary or a prophet. Benedict may be behind or ahead of his time, because one thing is certain – he certainly isn't of his time."

Difficult to pigeonhole

Mistrust of the zeitgeist is one of Benedict's outstanding intellectual characteristics, possibly a consequence of his having been born into a society that inflicted untold suffering on the world because it fell into unthinking conformity. Among his favourite passages from the scriptures is Romans 12:2: "Do not conform any longer to the pattern of this world, but be transformed by the renewing of your mind."

Like John Paul II, his position on several issues is more left than right. He condemned the invasion of Iraq. He deplores the death penalty as well as abortion. And the anti-free market stance of his most recent encyclical, on social and economic questions, came as an unwelcome revelation to his neocon admirers in the US. "This is not a conservative pope," insists Pera. "This is a pope who accepts the language of liberalism, in the original sense of the word, and wants to engage with all those who use that language. This is a pope who quotes Kant – a rationalist and Protestant, one of whose books was on the Index [of works banned by the Vatican]."

Pera himself illustrates why Benedict is difficult to pigeonhole. The few cardinals who have published books with lay writers have chosen unimpeachably orthodox Catholic journalists or academics. Yet Ratzinger teamed up with a former socialist who belongs to the party of Italy's divorced and secular prime minister, Silvio Berlusconi; a man who, when asked if he is an atheist, replies, "I can't say I'm a believer."

What brought him together with the future pope was a shared belief in one of the key ideas in Benedict's intellectual make-up: that many of the most treasured (and progressive) "western" values are inextricably bound up with Christianity and threatened by its decline. As Buttiglione puts it: "We are losing the values of the Enlightenment because we are abandoning Christianity and the idea of God." (Often the corollary of this view, which is widespread on the Italian right and centre-right, is that the biggest threat to western ideas comes from the advance of Islam.)

The idea that today's Catholic church – hostile to abortion and contraception, antipathetic to homosexuality and dismissive of the idea of women priests – might be a bastion of enlightened values is one many Britons will find hard to accept. But then it is difficult to think of a society that provides Benedict with a greater challenge to his ideas than multicultural Britain, with its anti-discriminatory ethos and increasingly vociferous atheist minority. The growing popularity of writers such as Richard Dawkins, Christopher Hitchens and AC Grayling, who argue that religion is not just irrelevant but pernicious, is a key difference between the Britain that Benedict will find when he visits and the altogether more respectful one John Paul II experienced 28 years ago.

The chances of yet another controversy during his visit are high, not least because – unlike his predecessor – Benedict writes all his keynote addresses himself. "You could get another Regensburg," warned the liberal Catholic who preferred not to be identified. "It is entirely possible."