The charge of hypocrisy, if proved, can destroy the career of any cleric, and Cardinal Keith O’Brien, who has died aged 80, was caught out spectacularly failing to practise what he had so vehemently preached. As the leader of the Catholic church in Scotland from 2003, he opposed very publicly, and in often intemperate language, all efforts to end discrimination against those in same sex relationships. Gay marriage, he said, was “grotesque”.

But then, with the media spotlight on him in February 2013 as the only British voter at the forthcoming conclave of cardinals to elect a new pope, it was revealed that O’Brien himself had had a homosexual relationship. His subsequent fall from grace was as spectacular as it was unprecedented. The Vatican, usually to be found denying such revelations and standing by senior clerics against all odds, effectively sacked O’Brien overnight and banned him from attending the conclave. (It turned out that they had known of the allegations against him since at least the previous year but, sadly typically, it was only when the charges were made public that action was taken.)

From denial of impropriety between 1965 and 2001 with four priests and seminarians, O’Brien first admitted in general terms to sexual conduct that had “fallen beneath the standards expected of me”. He was then revealed to have had an enduring relationship with one of the men in question right up to the time when he was taking to public platforms to denounce others in similar same-sex partnerships.

At first, O’Brien simply retreated to a retirement home in East Lothian, but his continuing presence in Scotland fuelled the row about his past behaviour. Allegations were made of cronyism, financial misconduct and a culture of secrecy. It was revealed by another Scottish bishop that, in 2012, O’Brien had blocked plans by his episcopal colleagues to commission an investigation into claims of child abuse by priests going back 60 years.

The Vatican stepped in and ordered O’Brien to leave the country. It sent in its own investigator, and appointed, Leo Cushley, a Motherwell-born Vatican official as O’Brien’s successor in the archdiocese of St Andrews and Edinburgh. To Cushley fell the task of repairing the damage done by the O’Brien scandal which, he said, had “distressed many demoralised faithful Catholics and made the church less credible to those who are not Catholic”. In March 2015, Pope Francis formally accepted O’Brien’s resignation, but allowed him to keep the title cardinal, while ordering him not to set foot in Scotland, nor to say mass or appear in public.

It is tempting to dismiss O’Brien as a prime example of the sort of double standards in matters of sexual morality that has for so long plagued and discredited a Catholic leadership fond of promoting a set of near-impossible standards in intimate relationships for the laity that has alienated so many of the faithful. And, indeed, he was. But O’Brien was not an out-and-out reactionary, nor was he always so out of touch with reality. The very day before he was outed, he had spoken up bravely, forcefully and sensibly against his Church’s cherished official line that married men could not be priests.

With hindsight, he may even have been making a veiled reference to his own private circumstances when he said: “It is a free world and I realise that many priests have found it very difficult to cope with celibacy as they lived out their priesthood and felt the need of a companion, of a woman, to whom they could get married and raise a family of their own.”

It was a remark that arguably chimed better with the humane Keith O’Brien that many had come to know and respect in the years before he became leader of the Scottish Catholic church. As the number two figure in the hierarchy north of the border from 1985, he had won a reputation in church circles and beyond as a thoughtful, grounded and occasionally outspokenly liberal figure.

The change in O’Brien’s public persona happened overnight. The Vatican’s announcement in 2003 that O’Brien, the archbishop of Saint Andrews and Edinburgh, was to become Scotland’s third cardinal since the Reformation, was inevitably followed by a round of interviews. In them, he reiterated what he had said many times before – that there needed to be more discussion within the church on contentious subjects such as clerical celibacy and contraception. He would welcome a fresh look at church teaching on the latter, he told reporters.

Within hours of these remarks being published, however, he was busy backtracking. During a routine service in St Mary’s Cathedral, Edinburgh, he surprised worshippers by reading out a personal “profession of faith” that, it was said, had been forced on him by the Vatican. In it he pledged 100% backing to the church’s ban on all artificial contraception and its insistence that priests be male and celibate.

O’Brien thereafter never returned to his previous concerns for dialogue and did not depart from the letter of church teaching in his public pronouncements, right up to his remarks on priestly celibacy on the eve of the papal conclave. And while he maintained his former genial ease when speaking in pulpits, on platforms or in TV studios, his message became ever more uncompromising and, some said, political.

In 2007, marking the 40th anniversary of the introduction of the Abortion Act, he warned pro-choice Catholic legislators that they could not regard themselves as full members of the church and provocatively (and insensitively, some said) described the termination rate in Scotland as equivalent to “two Dunblane massacres a day”.

His greatest political triumph came in 2008, when his strongly worded and high-profile intervention into the rapidly escalating clash between church and state over embryo experimentation forced a climbdown by the prime minister, Gordon Brown.

O’Brien, a former school science teacher, described proposals in the human fertilisation and embryology bill to allow scientists to create part-human, part-animal embryos for use in stem-cell experimentation, as “Nazi-style experiments” and “a monstrous attack on human rights, human dignity and human life”. Within days, the Labour government had capitulated and conceded what it had previously been refusing – a free vote on the most contentious clauses for MPs.

It caused O’Brien to be held up by traditionalist Catholics south of the border as a rebuke to their own leaders. The English and Welsh Catholic church, a separate entity in the eyes of the Vatican, had by contrast tended to adopt a softly-softly stance in the political arena, working increasingly behind closed doors in Whitehall and Westminster to raise concerns with ministers and anxious not to be seen as out on a limb. O’Brien’s success in forcing Brown into a U-turn may even have caused his English and Welsh counterparts to take a more robust stance, notably in opposing gay marriage, albeit to no avail.

The son of Alice (nee Moriarty) and Mark O’Brien, he was born in Ballycastle, County Antrim. His father, who had served with distinction in the Royal Navy during the second world war, was unable to find work on being demobbed, a result of the sectarianism in Northern Ireland workplaces at the time. The family was forced to emigrate to Scotland. The memory lived on with O’Brien who, as cardinal, tackled head-on what he called Scotland’s “secret sectarianism”, and attended Celtic-Rangers matches with other church leaders to try to calm the religious rivalry that traditionally accompanied such games. He even pushed – unsuccessfully – for the Act of Settlement to be amended to remove the bar on a Catholic monarch because it was a tacit endorsement of anti-Catholic prejudice.

O’Brien completed a degree in chemistry and mathematics at Edinburgh University and was ordained a priest in 1965. He initially combined being a curate with teaching in the local Catholic high school in Cowdenbeath, but from 1972 had his own parishes in first Kilsyth and then Bathgate. In 1978 he was named spiritual director of St Andrew’s College at Drygrange, and then in 1980 rector of St Mary’s College, the junior seminary (since closed) at Blairs. It was at these institutions that what he later referred to as “drunken fumblings” took place with younger men in his care.

In 1985, he was named to succeed Cardinal Gordon Grey as archbishop of St Andrews and Edinburgh. It was not until Grey’s death in 1993, however, that Rome had to name a new Scottish cardinal. While O’Brien had a good working relationship with his only other rival for the honour, his Glasgow counterpart, Archbishop Thomas Winning, the two were very different characters. Winning was a spiritual pugilist, a self-conscious prince of the church, a man of firm and traditional opinions, qualities which won him the nod from Rome.

Operating in Winning’s shadow had its advantages for O’Brien. In 1996, when Roddy Wright, the bishop of Argyll, was engulfed in scandal on account of his relationships with women, it was technically O’Brien’s responsibility to sort out the mess, but Winning took charge – and on this occasion the flak for the Church’s hypocrisy.

And in 1999, at a synod of European bishops, while Winning was patiently engaged in lobbying Vatican officials, O’Brien was able to win plaudits for coming clean in public about the degree of Rome’s control and interference in local churches. He told reporters, in relation to a joint pastoral plan of the Scottish and English bishops that Vatican officials had vetoed, that “we [bishops] know that we are vicars of Christ in our own dioceses, but the bishops here in Rome don’t think that”.

However, with Winning’s sudden death in 2001, O’Brien’s turn as cardinal had come – and with it the behaviour expected of those in this exclusive club of 120 or so international leaders of the Catholic church. He was not entirely orthodox. He celebrated his consecration in Rome as a cardinal, for example, by waving the Scottish flag in St Peter’s Square, a typically exuberant and relaxed gesture captured in a picture that graced many newspaper front covers around the world.

But in public, he ceased to express doubt or admit that anything other than the official church line had merit. He became a formidable political operator, helped by the fact that the new Scottish parliament was located in his diocese. He was courted in particular by Alex Salmond, leader of the Scottish Nationalists, who was anxious to break the traditional close link between Scotland’s Catholics and the Labour party. In 2006, O’Brien appeared to endorse aspirations for Scottish independence, though he later played down the remark.

In private, friends said, O’Brien remained the same man he had always been – laid-back, self-deprecating, funny – but those who had hoped this persuasive prelate might be a force for reform in the upper echelons of the church were disappointed.

He lived out his retirement in exile in Northumberland and then Newcastle, but his name will remain as a rebuke to the church’s sometimes overbearing claim to moral authority in all matters.

He is survived by his brother, Terry.

• Keith O’Brien, priest, born 17 March 1938; died 19 March 2018