What is it about a gold mitre, a flowing robe, a flash of cardinal red that so clouds our judgment? It is as if we believe these things hold a kind of magic. Don them and the wearer becomes pure and invincible. No human urges, no troublesome sexuality. Some people are naively enthralled by hierarchy. Priest, good. Bishop, better. Cardinal, best of all. The four complainants in the Cardinal O'Brien affair, who have accused him of inappropriate behaviour, haven't rated much sympathy within this strange moral hierarchy. "Who are they?" I have been asked all week. "Where are they?" has been another frequent question. But I have rarely been asked: "How are they?"

A narrative has begun to be embroidered on the cardinal's magic mitre. A fairytale. He is named but his accusers are not, and therefore the accusations are invalid. Let us be clear about one thing: the three priests, and one former priest, who have made complaints are not anonymous. They have given sworn, signed statements to the papal nuncio. The unnerving thing about the hunt to "out" these men (my phone has not stopped ringing with offers to "make it worth my while") is that it suggests people who have suffered traumatic events have no rights over how to tell their story, or how much information is made public. We demand not just that the appropriate authorities know names – we, the public, should know them, too.

In purely human terms, the story of Cardinal O'Brien's resignation is tragic. He had spent a lifetime reaching the upper echelons of his church, but after allegations of inappropriate behaviour made in the Observer last Sunday his fall from grace took just 36 hours. Not one of the four complainants takes any satisfaction from that. This is not about the exposure of one man's alleged foibles. It is about the exposure of a church official who publicly issues a moral blueprint for others' lives that he is not prepared to live out himself. Homosexuality is not the issue; hypocrisy is. The cardinal consistently condemned homosexuality during his reign, vociferously opposing gay adoption and same-sex marriage. The church cannot face in two directions like a grotesque two-headed monster: one face for public, the other for private.

There have been some misunderstandings about the timing of this tale: ridiculous accusations about the complaints spoiling the cardinal's retirement and having "the whiff of payback" for petty jealousies. Then it was suggested that this was all a conspiracy to prevent Keith O'Brien going to the conclave.

But in many ways this story was overtaken by events. The four complainants made their statements to the papal nuncio, Archbishop Mennini, around 8 or 9 February. On 11 February the pope resigned. The first response the complainants received from the nuncio said O'Brien should continue to go to Rome because "that will make it easier to arrange his retirement to be one of prayer and seclusion like the pope". The complainants recognised church subtext. In a message to me one wrote: "This is saying, 'leave it to us to sweep it under the carpet and you can forget about it. It will fade away as if we have dealt with it.' Not acceptable."

On 22 February, the cardinal gave an interview to the BBC about going to the conclave. He also said that church rules on celibacy should be reviewed. Informally, the men heard that the church was unhappy about that interview. Action would be taken. The cardinal would not go to Rome.

So did the church act because it was shocked by the claims against the cardinal or were they were angry he had broken ranks on celibacy? Two days later, the Observer published the story.

But why had the men waited so long to report allegations dating back to the 1980s? The answer is that people who have suffered trauma are not public property. They have the right to come to terms with it in their own time and express it in their own way, when they are ready. Being ready can simply be a collision of circumstances. Often, it's as straightforward as realising you are not the only one.

Sometimes as a journalist, you hold one piece of a jigsaw puzzle for a very long time. Gradually, you pick up another piece, and then another, until the picture clicks together and makes sense. I had known one element of this story for years: the former priest's. Let's call him Lenny. Now married, Lenny had been approached by the cardinal while a seminarian. Lenny says the cardinal was his spiritual director and used bedtime prayers as an opportunity to make advances to his young student.

"I knew myself to be heterosexual," he says, "but I did say to others that I thought it would be easier to get through seminary if you were gay."

Last month I received a call from Lenny. He was very shaken. He had had a conversation with a priest – we'll call him Peter – whom he hadn't spoken to for years. Peter told Lenny about an inappropriate relationship the cardinal had instigated with him. Two other priests were drawn in: Kenny and John. Both had experienced unwanted advances from the cardinal.

"I'd never wanted to 'out' Keith just for being gay," says Lenny. "But this was confirming that his behaviour towards me was part of his modus operandi. He has hurt others, probably worse, than he affected me. And that only became clear a few weeks ago."

Last week there were claims the cardinal did not know details of the allegations. How could he respond, the implication was, if he did not know what he was being accused of? That was simply untrue. Last Saturday, the day before the Observer printed the story, the cardinal did not respond to calls and messages left for him. The Scottish Catholic Media Office was approached. Peter Kearney, the communications director, asked for the allegations to be put in writing. They were. In that email, four separate allegations were outlined. At the end, a direct question was posed: "Is it true that the cardinal has broken his vow of celibacy?" The allegations could not have been more specific.

Kearney certainly seemed to understand at the time. His response was brief: "The cardinal is consulting his lawyers. These claims are contested and should not be published." But I had four statements that described the cardinal attempting to touch, kiss, or have sex with people in his care.

"He started fondling my body, kissing me and telling me how special I was to him and how much he loved me," one had written. One of the statements was five pages long. Given the strength of the evidence we had, the Observer chose to publish the story.

There have been many questions about the four complainants that cast doubt on them and their motives. So let me tell you about the men I have come to know. They are men of conscience and integrity who desperately want to do "the right thing". Men who love the church but recognise that the way it covers up scandal and hides wrongdoing is damaging. On a personal level they are funny, kind, spirited, generous, conventional and unconventional in different measures. But above all they are brave. Peter wrote to me saying it had been the worst week of his life. He couldn't eat, couldn't sleep. Each of those men spoke out knowing it could ruin their lives. Some of them were trying to work out what order they might be able to take refuge in if the church disowned them for speaking.

The biggest sin in the Catholic church has historically been "scandalising the faithful". That is why the abhorrent cover-ups of child sex-abuse scandals have been part of the church's history. They shield their own – and if you speak against them, you stop being their own. Archbishop Tartaglia of Glasgow – who caused outrage last year when he linked the tragically premature death of David Cairns MP to his homosexual lifestyle – publicly said prayers for the cardinal at mass in Edinburgh after being named as the cardinal's temporary replacement. He invited the cameras in while he did it. It is right that the cardinal is given adequate support. It is not right if the church pretends that he is the victim in this. The gold mitre, the cardinal's robes, do not make him more worthy of support than the men in ordinary clerical collars.

It seems there is a great deal of displacement activity going on in the Scottish Catholic Church. It is not the behaviour of the four complainants that should be concentrated on. It is the behaviour of the cardinal. How big a crisis this is for the church lies in its own hands. The signs so far do not suggest a new era of openness. But, as the church itself proclaims, redemption is always possible for a sinner.

Priests tell me there is a "gay culture" in the Scottish Catholic church – but not an open, healthy one. In some ways, perhaps it shouldn't be a surprise. The church has always had a deeply cynical side when it comes to sexual morality. Lenny recalls being a young priest, accompanying an older priest who would rise to great heights in the church. The older man was drunk and was ranting about men who left the priesthood. Why leave to have sex? Why didn't they just visit a sauna and go to confession in the morning?

A cardinal does not resign overnight over trivia. Some people have questioned, though, whether his alleged behaviour constitutes abuse. After all, this involves adults, not children. One commentator even suggested it's all just a scandalous homophobic plot. That completely misunderstands the nature of the power a spiritual director has over his seminarians and a cardinal has over his priests. Lenny gave up his priesthood when O'Brien was promoted to be his bishop. He did not want to be in his power. "He harmed me in so many ways," he explained.

And ask Peter if this story involved abuse. Peter has undergone long-term psychological counselling. His experiences with the cardinal are part of his records. Peter admits he even contemplated suicide. And still people are shouting "Reveal yourself!"

Why should he?

A few nights ago Lenny had a dream. He and his fellow complainants were in a cold, damp church, searching for a piece of scripture for a funeral. The Bible they were looking in was tattered. They could not find the words. When he woke, Lenny knew exactly the passage they had been hunting for: Ecclesiasticus 2. He wants the words read at his own funeral, to be acknowledged in the end as a priest.

"My son, if you aspire to serve the Lord,

Prepare yourself for an ordeal…

…Since gold is tested in the fire

And chosen men in the furnace of humiliation."

There is the superficial gold of the mitre, and then there is solid gold. The church has to learn the difference. When Lenny told the others his dream, one said he, too, had dreamed about their situation. His dream had been simpler. Keith O'Brien had asked their forgiveness for his behaviour. All of them had granted it.