It started out as a well-intentioned attempt to bring festive cheer to some of society's most neglected members – the hundreds of children who each year are caught up in the UK's asylum system.

But when the Anglican church's leading expert on Father Christmas, dressed as St Nicholas himself, arrived with one of Britain's most distinguished clerics to distribute presents to children held at the Yarl's Wood immigration removal centre in Bedfordshire, things took a turn straight out of Dickens.

An unedifying standoff developed that saw the security personnel who guard the perimeter fence prevent St Nicholas, the patron saint of children and the imprisoned, from delivering £300 worth of presents donated by congregations of several London churches.

In a red robe and long white beard, clutching a bishop's mitre and crook, St Nick – in real life, the Rev Canon James Rosenthal, a world authority on St Nicholas of Myra, the inspiration for Father Christmas – gently protested that he was not a security threat, but to no avail.

Then as St Nicholas, accompanied by the Rev Professor Nicholas Sagovsky, canon theologian at Westminster Abbey, attempted to bless the gifts, the increasingly angry security guards called the police. The resulting ill-tempered and surreal impasse between church and state was videotaped by asylum seeker support groups and could become an internet viral hit.

The row comes amid mounting concern about the treatment of children in immigration removal centres. Last week senior doctors called for an immediate end to the "profoundly harmful" detention of children in immigration removal centres. In today's Observer a number of leading children's authors – including Michael Morpurgo, Michael Bond and Philip Pullman – have signed a letter calling for an end to child detention.

Rosenthal, who regularly appears alongside the Archbishop of Canterbury at festive parades, is the founder of the St Nicholas Society, which was set up to promote "interest, learning, and appreciation of the tradition of St Nicholas".

"St Nick has never been turned away from anywhere before," Rosenthal said. "So I was extremely disappointed not to be able to hand deliver the gifts to the children detained at Yarl's Wood. I hope the kids realise that they will be firmly in my prayers."

The St Nicholas Society, along with Citizens for Sanctuary, which campaigns to end the detention of children and families in the asylum system, is writing to the centre's management to complain at how it handled the pre-announced visit. They have complained about the heavy-handed tactics employed by the guards who patrol the perimeter fence and Serco, the private company that operates Yarl's Wood.

The two groups say that Serco refused requests to provide details about the 35 children in the centre so they could receive appropriate presents. They complain that the company did not respond to numerous requests to discuss how a handover of presents could be carried out if St Nicholas was prevented from entering during his visit this month.

Serco also refused permission for the two clerics to enter the centre to visit two refugee families later the same day, as it had previously agreed. They were handed letters from Dawn Elaine, contracts manager at Yarl's Wood, saying permission had been revoked because of "concerns about your conduct".

The minor row is threatening to escalate into a bigger furore over the government's policy of keeping children in Immigration Removal Centres.

"If this is how visitors are treated, I shudder to imagine what else transpires inside Yarl's Wood," Rosenthal said.

Sagovsky added: "This was about bringing a moment of joy to kids locked up in a deplorable situation. I can't help but contrast the smiles and wonderment on the faces of the children St Nicholas visited at a local primary school with the sad fate of those kids who will be locked up in Yarl's Wood over Christmas."

It is estimated that 1,000 children are detained every year. Last week the Royal Colleges of Paediatrics and Child Health, alongside leading GPs, psychiatrists and the UK Faculty of Public Health, warned the detention of children and their families caused "significant harm" and should be ended without delay.

Dr Philip Collins, a forensic adolescent psychiatrist at the South London and Maudsley NHS Foundation Trust, said children of asylum seekers were "uniquely at risk" of "very high levels" of mental health problems.

The organisation Medical Justice said it had seen more than 100 children in Yarl's Wood Immigration Removal Centre, and shared the doctors' concerns. The organisation's clinical director, Dr Frank Arnold, said: "Our findings accord exactly with those of the royal colleges.

Over the years, when challenged, the response of UK Borders Agency (UKBA), and the private company it subcontracts healthcare to, have ranged from pleading ignorance, to painting walls a different colour, to publishing a vast number of documents, which prove meaningless because the health outcomes for children seem no better, and remain frightening."

A spokesman for Serco referred questions to the Home Office, which insists only people subject to stringent security checks can be allowed into the detention centre and there can be no exceptions.

However, it seems even the Home Office is keen to avoid being labelled Scrooge. Alan Kittle, Director of Detention Services for the UK Border Agency said: "UKBA already has procedures in place to ensure any children held in detention over the Christmas period receive a present and get to see Santa."