Cardinal Keith O'Brien, the UK's most senior Roman Catholic cleric, has resigned as the head of the Scottish Catholic church after being accused of "inappropriate acts" towards fellow priests.

News that Pope Benedict had accepted the cardinal's resignation as archbishop of St Andrews and Edinburgh came after the Observer disclosed a series of allegations by three priests and one former priest.

O'Brien has denied the allegations and had been expected to continue in his post as archbishop until mid-March, when he was due to retire at age 75.

However, in a statement released by the church on Monday, it emerged that the pope had accepted O'Brien's resignation a week ago, on 18 February.

In the statement, O'Brien apologised to any people he had let down and said he did not want the controversy to overshadow the election of the new pope.

"I have valued the opportunity of serving the people of Scotland and overseas in various ways since becoming a priest," he said. "Looking back over my years of ministry, for any good I have been able to do, I thank God. For any failures, I apologise to all whom I have offended."

His resignation means the cardinal will not now take part in the election of a successor to Pope Benedict. This will leave Britain unrepresented in the process, as O'Brien was the only cardinal in the British Catholic churches with a vote in the conclave.

Although Cardinal Cormac Murphy-O'Connor, the archbishop emeritus of Westminster and former leader of Catholics in England and Wales, will attend pre-conclave meetings, he will not have a vote in the election itself as cardinals aged 80 and over are ineligible to vote. He is 80.

O'Brien, who missed celebrating mass at St Mary's Cathedral in Edinburgh on Sunday, had been due to fly out to the Vatican on Tuesday for the conclave.

His resignation is a heavy blow to the church and Benedict, whose papacy has been beset by repeated controversies over misconduct by clergy in Europe and the US and allegations of corruption and incompetence at the Vatican.

However, with the Vatican and Benedict's successor facing a series of serious challenges to its reputation, O'Brien's speedy retirement will allow the church to move quickly to settle this controversy.

The Observer reported that the four men came forward last week to demand his resignation largely because the complainants did not want O'Brien taking part in the papal election.

O'Brien said he had already agreed with Benedict that he would step down on 17 March as he was "approaching the age of seventy-five and at times in indifferent health". The pope had now agreed he could resign immediately, he said, forcing the church to find an "apostolic administrator" to run the diocese until a new archbishop could be appointed.

Confirming he would not now go to the conclave, O'Brien said: "I thank Pope Benedict XVI for his kindness and courtesy to me and on my own behalf and on behalf of the people of Scotland, I wish him a long and happy retirement.

"I also ask God's blessing on my brother cardinals who will soon gather in Rome to elect his successor. I will not join them for this conclave in person. I do not wish media attention in Rome to be focused on me, but rather on Pope Benedict XVI and on his successor. However, I will pray with them and for them that, enlightened by the Holy Spirit, they will make the correct choice for the future good of the church.

"May God, who has blessed me so often in my ministry, continue to bless and help me in the years which remain for me on Earth and may he shower his blessings on all the peoples of Scotland especially those I was privileged to serve in a special way in the archdiocese of St Andrews and Edinburgh."

Scotland's first minister, Alex Salmond, said he had learned of the cardinal's decision with "the greatest sadness".

He said: "In all of my dealings with the cardinal, he has been a considerate and thoughtful leader of the Catholic church in Scotland, stalwart in his faith but constructive in his approach.

"The hugely successful visit of Pope Benedict in 2010 was a highlight of his cardinalship and symbolised the key role of the Catholic church in Scottish society."

Salmond said it would be a "great pity if a lifetime of positive work was lost from comment in the circumstances of his resignation". He added: "None of us know the outcome of the investigation into the claims made against him but I have found him to be a good man for his church and country."

Austen Ivereigh, a Catholic writer and co-ordinator of Catholic Voices, said he was not surprised that the church had moved so quickly following the emergence of the allegations.

"I think the speed of the announcement has everything to do with the fact that these accusations were made on the eve of the papal election," he said. "It was important not to distract from the pope and the election process, and I think frankly it was a necessary act and [O'Brien] did it for the good of the church."

Ivereigh said the rapid response showed both the church's "renewed transparency and accountability" and its desire for the election of Benedict's successor to proceed as uncontroversially as possible.

He described O'Brien as a "very affable, warm and hospitable" man who was always unafraid to speak his mind. "He's never been considered one of the high-flying cardinals; he doesn't know Rome that well or have fluent Italian and so he's never been a cardinal who has been as significant in the Vatican as Cardinal Cormac Murphy-O'Connor," said Ivereigh.

"But he's been a very stalwart defender of the Scottish church's stances on various issues and he has been valued for his forthrightness and directness – even though I think sometimes some of his pronouncements have not been judiciously phrased."

Ivereigh also pointed out that although O'Brien's decision not to attend the conclave left British Catholics without a vote in the election of the next pope, it did not leave them without a voice.

"It's important to remember that it's not just the conclave where cardinals bring their influence to bear," he said. "In many ways the more important time over the next few weeks will be the general congregations when the cardinals meet together before the conclave to discuss the state of the world and the state of the church – and Cardinal Murphy-O'Connor will be present at those because the over-80 cardinals are part of those discussions; they're just not allowed to vote.

"I think the perspectives of the British church will still make themselves felt within the college because of Cardinal Murphy-O'Connor's presence and influence."

O'Brien has been an outspoken critic of gay rights, denouncing plans for the legalisation of same-sex marriage as "harmful to the physical, mental and spiritual wellbeing of those involved". He was named bigot of the year in 2012 by the gay rights group Stonewall because of his central role in opposing gay marriage laws in Scotland.

Colin Macfarlane, the director of Stonewall Scotland, called for a full inquiry into the claims against the former cardinal. "We trust that there will now be a full investigation into the serious allegations made against ex-cardinal O'Brien," Macfarlane said. "We hope that his successor will show a little more Christian charity towards openly gay people than the former cardinal did himself."