Three priests and a former priest in Scotland have reported the most senior Catholic clergyman in Britain, Cardinal Keith O'Brien, to the Vatican over allegations of inappropriate behaviour stretching back 30 years.

The four, from the diocese of St Andrews and Edinburgh, have complained to nuncio Antonio Mennini, the Vatican's ambassador to Britain, and demanded O'Brien's immediate resignation. A spokesman for the cardinal said that the claims were contested.

O'Brien, who is due to retire next month, has been an outspoken opponent of gay rights, condemning homosexuality as immoral, opposing gay adoption, and most recently arguing that same-sex marriages would be "harmful to the physical, mental and spiritual well-being of those involved". Last year he was named "bigot of the year" by the gay rights charity Stonewall.

One of the complainants, it is understood, alleges that the cardinal developed an inappropriate relationship with him, resulting in a need for long-term psychological counselling.

The four submitted statements containing their claims to the nuncio's office the week before Pope Benedict's resignation on 11 February. They fear that, if O'Brien travels to the forthcoming papal conclave to elect a new pope, the church will not fully address their complaints.

"It tends to cover up and protect the system at all costs," said one of the complainants. "The church is beautiful, but it has a dark side and that has to do with accountability. If the system is to be improved, maybe it needs to be dismantled a bit."

The revelation of the priests' complaints will be met with consternation in the Vatican. Allegations of sexual abuse by members of the church have dogged the papacy of Benedict XVI, who is to step down as pope at the end of this month. Following the announcement, rumours have swirled in Rome that Benedict's shock move may be connected to further scandals to come.

The four priests asked a senior figure in the diocese to act as their representative to the nuncio's office. Through this representative, the nuncio replied, in emails seen by the Observer, that he appreciated their courage.

It is understood that the first allegation against the cardinal dates back to 1980. The complainant, who is now married, was then a 20-year-old seminarian at St Andrew's College, Drygrange, where O'Brien was his "spiritual director". The Observer understands that the statement claims O'Brien made an inappropriate approach after night prayers.

The seminarian says he was too frightened to report the incident, but says his personality changed afterwards, and his teachers regularly noted that he seemed depressed. He was ordained, but he told the nuncio in his statement that he resigned when O'Brien was promoted to bishop. "I knew then he would always have power over me. It was assumed I left the priesthood to get married. I did not. I left to preserve my integrity."

In a second statement, "Priest A" describes being happily settled in a parish when he claims he was visited by O'Brien and inappropriate contact between the two took place.

In a third statement, "Priest B" claims that he was starting his ministry in the 1980s when he was invited to spend a week "getting to know" O'Brien at the archbishop's residence. His statement alleges that he found himself dealing with what he describes as unwanted behaviour by the cardinal after a late-night drinking session.

"Priest C" was a young priest the cardinal was counselling over personal problems. Priest C's statement claims that O'Brien used night prayers as an excuse for inappropriate contact.

The cardinal maintained contact with Priest C over a period of time, and the statement to the nuncio's office alleges that he engineered at least one other intimate situation. O'Brien is, says Priest C, very charismatic, and being sought out by the superior who was supposed to be guiding him was both troubling and flattering.

Those involved believe the cardinal abused his position. "You have to understand," explains the ex-priest, "the relationship between a bishop and a priest. At your ordination, you take a vow to be obedient to him.

"He's more than your boss, more than the CEO of your company. He has immense power over you. He can move you, freeze you out, bring you into the fold … he controls every aspect of your life. You can't just kick him in the balls."

All four have been reluctant to raise their concerns. They are, though, concerned that the church will ignore their complaints, and want the conclave electing the new pope to be "clean". According to canon law, no cardinal who is eligible to vote can be prevented from doing so.