March 13, 2013. The world is waiting. Television screens show days-old footage of cardinals in red and white, processing past Vatican guards into the magnificence of the Sistine Chapel for the papal conclave. Every image, from the polished marble floors and gold ceilings to the priceless frescoes on the walls, tells a story of wealth, pageantry and power. Outside, in St Peter's Square, the crowds are cheering for a man whose name they do not yet know. But there is another soundtrack. The day before, Pat McEwan, a 62-year-old from Scotland, had described to me how he was raped at the age of eight by a priest. His voice drowns out crowds and choirs. "I ran home shaking like a dog. I had wee short trousers on and the shite was running down my leg. My mum and my auntie had to wipe me down."

The juxtaposition of those two images: the powerful institution that represents 1.2 billion Catholics and the abused child, tells the story of a church with two faces: one public and one private. Last month, the church was plunged into crisis when the Observer revealed that three priests and one ex-priest had complained to the Papal Nuncio about Cardinal Keith O'Brien, Archbishop of St Andrews and Edinburgh. The cardinal, who publicly decried homosexuals as degenerate, had, they said, privately been making advances to his own priests for years. But the story was never about one man. It wasn't about personal weakness. Keith O'Brien was merely a symptom of a wider sickness: an institution that chooses cover-up as its default position to conceal moral, sexual and financial scandal.

This was not paedophilia but it was an abuse of power – a man in authority acting inappropriately to young seminarians and priests under his control. It was made clear that a full sexual relationship had been involved. Yet there were attempts to cloud his behaviour in moral ambiguity. First, there was denial. The cardinal "contested" the allegations. A day after publication, he resigned. The next week, he issued a statement admitting his sexual conduct "as a priest, a bishop and a cardinal" had fallen short. Many ignored what that confirmed about the extent and duration of his behaviour: he was made cardinal in 2003.

Next, came obfuscation, with the church claiming it did not know the substance of the allegations, despite being given written notice before publication. Then, anger and the minimising of wrongdoing – the cardinal had been destroyed for mere "drunken fumblings" from 30 years ago. Why, he had probably been to confession and received absolution. But most revealing of all was the attempt to turn the spotlight on the complainants' motivation, to blame the accusers rather than the accused. It has been a familiar pattern in Catholic abuse cases over the years.

The stories you are about to read will take you from the late-1950s to the present day, a sweep of more than 50 years. Society has changed radically in those years, from the black-and-white morality of the 1950s, tenement slums and rag-and-bone men, to the fast-living, flat-screen, iPhone generation of 2013. And yet, through all those decades, all those changes, the behaviour of the Catholic church towards abuse victims has changed remarkably little.

Two concepts are critical to understanding church behaviour. The first is "scandalising the faithful". Traditionally, the hierarchy believed the greatest sin was shaking the faith of Catholic congregations. Protecting them meant concealing scandal. Adopting that as your moral standpoint means anything goes. You can cover up sexual misconduct from those you demand sexual morality from. You can conceal financial corruption from those who put their pounds in the collection plate. You can silence the abused and protect the abuser. Guilt about sacrificing individuals is soothed by protecting something bigger and more significant – the institution.

The second concept is "clericalism", a word used to describe priests' sense of entitlement, their demand for deference and their apparent conformity to rules and regulations in public, while privately behaving in a way that suggests the rules don't apply to them personally. (O'Brien was, in that sense, a classic example.) The Vatican is an independent state; the Holy See a sovereign entity recognised in international law and governed by the Pope. The Nunciature operates like government embassies in different countries worldwide. It is even governed by its own rules: Canon Law. All this contributes to the notion that the church can conduct its own affairs without interference or outside scrutiny. It demands a voice in society without being fully accountable to it.

In the weeks following O'Brien's departure, several priests' meetings were held in his diocese. One was chaired by his temporary replacement, Archbishop Philip Tartaglia of Glasgow, and O'Brien's auxiliary bishop, Stephen Robson. Some priests wanted messages of support sent to the cardinal, encouraging him to return to Scotland for his retirement. Compassion for a sinner? Or clerical cover-up? Some not only knew of the cardinal's behaviour, they may have been subject to it.

"The clerical power structure not only protects clergy who are sexually active but sets them up to live double lives," says Richard Sipe, an American psychotherapist and ex-priest who has spent many years researching celibacy and abuse. "Corruption comes from the top down. Superiors, rectors and bishops do have sexually active lives and protect each other – a kind of holy blackmail."

Is this the biggest crisis for the Catholic church since the Reformation, asked Professor Tom Devine, one of Scotland's leading historians? But one cardinal is not the crisis. Thousands of abused children around the world, and an institution that silences them: that is the real crisis. The church claims child-protection policies have been in place in Scotland since 1999. Judge them for yourself in the following stories. Events come right up to the last few weeks, with Keith O'Brien's resignation as backdrop. The American civil-rights activist, Martin Luther King, once said, "There comes a time when silence is betrayal." In the Catholic church, that moment has long since passed.

Speaking publicly for the first time, Pat McEwan says he fell prey to a paedophile ring of priests. His main abuser, his parish priest, encouraged Pat to visit him, then appeared to slip into a trance. Pat shook him. "I've just been talking to Jesus and he says would you like to go to heaven?" said the priest. Then he asked, "Do you love your mummy?" Yes Father. "Do you love your daddy?" Yes Father. "Do you love me? Because this is our little secret and you mustn't tell your mummy or daddy or you will go to the burny fire."

This was the 1950s. Parish priests were honoured guests in Catholic homes. The priest arranged for Pat's devout mother to visit Carfin Grotto, leaving Pat with a priest friend of his. Pat remembers watching through the window while his mother disappeared into the grotto. As soon as she did, the priest turned to him. "I want you to do for me what you have done for your parish priest," he said. Then he raped him. Afterwards, he tried to quieten the child's tears before his mother returned. "God doesn't like boys who cry. Be a soldier of Christ."

Child abuse is rarely contained within childhood. The events bleed into every aspect of adult choices, relationships, employment and health. Victims suffer from alcoholism, mental-health issues and post-traumatic stress disorder. It is not uncommon for male victims to end up in prison. Cameron Fyfe is a Scottish lawyer who has dealt with more than 1,000 Scottish cases of abuse by the Catholic church. "Not one person has come out unharmed," he says. "Every one has had their life smashed." Pat is no different. He became an alcoholic, though he has now been sober for 18 months.

Pat approached the church in the late-90s. He never once asked for money. Instead, he sought counselling, a spiritual retreat – and acknowledgement. "This has always been about justice." He enlisted the support of Alan Draper, a child-protection expert who had worked for the church in the mid-90s. Draper had left, unhappy with the bishops' persistent refusal to take appropriate action. Now, he accompanied Pat to a meeting with Bishop Joseph Devine of Motherwell. In their accounts both Pat and Draper say that the bishop's solution to the horrifying tale was simple. "Pat, he's an old man," he said. "Please let him away with it."

Pat produces a file of letters, not just from the bishop but from his safeguarding team. The tone is frequently hostile, as if "safeguarding" in the diocese is not so much about protecting victims as protecting the church from victims. In one, Pat is berated for telephoning the office. "Could I please ask," writes diocesan safeguarding adviser Tina Campbell, "that if you wish to make contact with any member of the diocesan safeguarding team, this is done by letter and not on the phone?"

In 2010, Pat approached O'Brien. Despite being the most senior Catholic in Britain, O'Brien said he could not interfere in Bishop Devine's area. Draper subsequently wrote to Devine on Pat's behalf in February 2011, asking him to meet them both. He refused. Pat, he insisted, should meet him alone. "If he were to be accompanied by yourself or anyone else, the meeting would be cancelled," he wrote. "I take it that I have made myself clear to you on this matter." At the meeting, Devine rounded on Pat. "You are nothing but an alcoholic," he said.

"All Pat wanted," says Draper, "was for the bishop to say, 'Sorry, we believe you.'" In November last year, Pat finally received a letter from Tina Campbell saying that in "an attempt to bring some sort of closure" they were referring the case to Motherwell police, who are currently investigating. Pat's main abuser is now dead, but one remains alive. It has been a long journey.

The reality of "safeguarding" in the Catholic church is that each bishop presides over an independent fiefdom. Draper has asked for evidence of annual reviews that the church agreed to back in 1996. So far, they have not been forthcoming. In response to questions regarding church procedures in abuse cases, the Catholic church's director of communications, Peter Kearney, told the Observer, "'The church' as referenced in your question doesn't actually have a locus in this issue, in that in Scotland, 'the church' consists of eight separate and autonomous diocese, each with its own bishop and each responsible for the issue of safeguarding in their own area. The way a complaint is handled in one diocese should be the same as in every other, but… that hasn't always been the case."

It confirms, says Alan Draper, what he has been saying for years. "The bishops exercise tight control and do nothing for victims. The so-called national co-ordinator is effectively sidelined into training the laity and is toothless to do anything that really matters. It is a sham."

Ann Matthews also lives in Bishop Devine's diocese. In the 1980s, she was regularly abused from the age of 11 to 17 by her priest. She has never told her parents. They were extremely devout and the priest frequently said prayers in their house. After visiting Ann's dying grandmother, he came downstairs and tried to have sex with her on the sofa.

After accepting the abuse had happened, Devine quietly sent the priest away for counselling, telling the parish he was retiring due to ill health. That, says Ann, denied other parents the opportunity to assess whether their children had also been affected. Some studies suggest abuser priests may have around 50 victims.

Ann says her life has been broken. She suffers from eating disorders, sleep disorders, anxiety and depression. She is frequently suicidal. She has no job. She has a partner, but will never have children as she doesn't want to inflict her insecurities on a child. "Sometimes, it feels like I died a long time ago, that there's this body that walks around the earth and doesn't know it should lie down."

In a meeting that included priests of the diocese, she was asked why she allowed the abuse to continue. But Ann was a child. She tried to convince herself abuse was love. "I said to them, I am sitting here as a grown woman, but when this happened I had knee-high socks and bobbles in my hair." "Oh come on!" retorted one of the priests, before adding, "Give her money and let her run."

She never received money, but she did get counselling, which she was grateful for. In the next 12 years, the church never once asked for a report. Last year, they wrote out of the blue, telling Ann her funding was being withdrawn. Her final session would be May 2013. Her counsellor wrote to the church saying Ann has been suicidal for substantial periods and still needs support. "It's as if they calculated that I was abused for seven years," says Ann, "but had counselling for 12 – so time up. I'm just someone who has had a vast claim on their resources."

On 11 February, the day Pope Benedict resigned, Ann attended a meeting with safeguarding officer Tina Campbell regarding the termination of her counselling. She was accompanied by her psychotherapist and an advocacy worker. But the behaviour towards her was so hostile that she quickly fled in tears. The advocacy worker confirms she had to intervene because the church's behaviour was so unacceptable. An appeal was lodged and they were informed it would be held in Edinburgh. Ann has since received a letter saying that, "due to the complex situation in the Diocese of St Andrews and Edinburgh", no appeal can go ahead. Now, she waits.

The church has no policy regarding counselling. Again, individual bishops decide. Helen Holland was a victim of serious physical and sexual abuse in the 1960s and 70s in Kilmarnock's Nazareth House. As a child she was hooded, held down by a nun and raped by a priest. She went on to become a nun herself, but eventually left her order. Now vice-chair of the Scottish survivors' group, Incas, she has spoken on behalf of victims in the Scottish parliament.

Helen Holland, who was sexually assaulted during childhood by a nun at Nazareth House in Kilmarnock. Photograph: Murdo Macleod for the Observer

The legacy of her abuse is still with her and Helen has paid for counselling at different periods in her life. But in recent years, she started experiencing "night terrors", regularly sleepwalking outside her home. "It's like being a child all over again. My counsellor said I was trying to reach the child within and I said that little girl Helen died. She doesn't exist any more. But it's not as simple as that. I can't put the lid back on it."

Now on disability allowance because of ill health, Helen could no longer afford counselling. She wrote to the church last June, asking for help. She never received a reply. The nun who abused her was Irish so she made an application to the Irish government. It now funds her treatment rather than the church.

Charles Simpson, an Edinburgh man who says he was abused and raped by his parish priest in the 1990s, also ran into a church wall of silence. Charles had alcohol and drug problems following the abuse, and ended up in prison for continually breaking into the parish house where it had happened. "I was hitting back at the church. It was an angry time in my life." He is still on antidepressants and methadone. "I want to be able to function, to be a member of society, but it's hard. He had me so wrapped up in fear and loneliness, telling me my family was poor because they were unemployed. The things he said made me feel I had no strength."

Charles sought the help of a priest who approached O'Brien on his behalf. "The priest was told to keep quiet," says Charles, who subsequently asked the church for counselling. He, too, got no reply. The silence prompted him to take legal action: he is now suing the Archdiocese of St Andrews and Edinburgh for £100,000. His lawyer, Cameron Fyfe, says the church's official defences in the action have been surprising. For the sake of a legal defence, they have denied that one of their objectives is to "spread the word of God". And they have claimed they had no power to move or remove the priest, or to control – or even direct – his activities.

The time bar rule in Scottish law means civil action should be taken within three years of either the abuse, or the victim's 16th birthday. Most civil cases against the church have failed for that reason. Fyfe hopes the court will use its discretion to allow this case to proceed, but the process could take years. "Money…" says Charles wearily. "It doesn't change what happened. I feel like I'm up against it. To me, they are just legal gangsters."

In the wake of the O'Brien scandal, Archbishop Tartaglia, said – as if it were a rare accusation – that the most "stinging charge" against the church was hypocrisy. Yet the hierarchy knows further scandal is only a whisper away. The four complainants against the cardinal were accused of being part of a gay cabal. They were not. But priests and church insiders say a gay culture does exist in the Scottish church. This is about cronyism, secrecy and an all-male culture. The Scottish church still bears the scars from Roddy Wright, bishop of Argyll and the Isles, who ran off with a woman in 1996. Until O'Brien's behaviour was revealed, it was perhaps tempting for the hierarchy to believe gay priests were "safer". Homosexual affairs – especially with other clergy – are easier to hide than those involving women and children.

Homosexuality is only an issue because of the church's public stance on it. It should go without saying that there is no link with abuse. But Richard Sipe believes there may be a link between abuse and celibacy. In 1990, he published a 25-year American study showing that at any one time, 50% of priests will have been sexually active in the past three years. That figure has been replicated in other places: Spain, Holland, Switzerland and South Africa. "O'Brien and Scotland are not alone or exceptions," says Sipe.

The Catholic church has created a hierarchy of sexual morality with celibacy at the pinnacle. But that can create distortions. Sipe's studies suggest around 70% of priests display psychosexual immaturity. Celibacy, he argues, is not something most people can achieve. When legitimate sexual outlets are forbidden, some turn to illegitimate ones. "The majority of clergy are unable to deal with sexual deprivation in healthy ways," he argues. Around 6% of priests will have sex with minors. In Australia, abuse by Catholic priests is six times higher than other churches combined.

David has direct experience of Australia and New Zealand. He rebuffed the sexual advances of a 65-year-old Jesuit in New Zealand when he was 14. He later joined the religious life himself and was sexually approached both in a Cistercian order and a seminary. In Australia, he was approached by a senior priest in a Dominican priory. Many priests have similar stories, but keep quiet because they are still part of the institution. David, however, left the religious life.

Afterwards, he had an affair with a man he calls Peter, who had left a seminary in Rome. Peter took David to his old haunts, calling in on a convent he had visited for weekly confession. His confession was always heard last, after the nuns, by a priest who later became a bishop. "At the top of the convent," says David, "there was a comfortable room set aside for confession. But what started as confession turned into a weekly lover's tryst. Peter, who was somewhat bitter about having quit Rome, was eager during that holiday to tell me the exact nature of their lovemaking. It involved anal intercourse." The priest – whom David names – was operating at the highest levels of the Vatican.

There were those who tried to make O'Brien into a victim. Perhaps he was a victim of a dysfunctional system. But the real victims are the powerless and voiceless. Many live lives they feel are tainted and will never wash clean. Michael is an ex-seminarian who went to the police when O'Brien refused to take appropriate action against his abusers in seminary. Known in the Scottish press as "Michael X", he eventually received £42,000 compensation from the Catholic church, which Sipe estimates has paid out £3bn worldwide.

Michael has previously described how he told his spiritual director about the abuse. The man assured him he was not to blame – then made sexual advances, too. What Michael hasn't revealed before is his guilt at what happened next. He had to serve on the altar for the spiritual director at a private mass. "At the prayer, 'Lord have Mercy'," Michael recalls, "he dropped to his knees and grabbed my legs. He was shaking from head to toe, saying, 'Lord have mercy, Michael have mercy.' It was horrendous. He disintegrated in front of me." The priest died of a brain haemorrhage not long after. When it was suggested the cause was stress, Michael felt devastated.

Many shoulder the guilt and shame that belongs to their abusers. Ann cannot let go of that question, "Why didn't you do something?" In an email after we talk, she writes: "I am not sure how much longer I can go on. The sad thing is that even if I ended my life, I would simply become another statistic."

Crisis always provokes choice: to go on in the same direction or to change course. When Martin Luther King talked of the betrayal of silence, he said decisions had to be made. "If we but make the right choice," he continued, "we will speed up the day… all over the world, when justice will roll down like waters, and righteousness like a mighty stream."

Some names have been changed