One of the pope's top advisers on his visit to England and Scotland has dropped out of his entourage following the publication of an interview in which he said that arriving in Britain "you sometimes think you've landed in a third world country".

Benedict XVI's spokesman, Father Federico Lombardi, told the Guardian, however, that Cardinal Walter Kasper had withdrawn "for health reasons".

He said the 77-year-old prelate's absence from the papal party, which lands in Edinburgh tomorrow at the start of a four-day visit, "had absolutely nothing to do with anything else".

Kasper, the Vatican's leading expert on relations with the Church of England, made his remark after noting that Britain was a "secular, pluralistic" country.

Asked by the German news magazine, Focus, whether Christians were discriminated against in Britain, he replied: "Yes. Above all, an aggressive new atheism has spread through Britain. If, for example, you wear a cross on British Airways, you are discriminated against."

His comment on the airline is bound to revive questions about why, in a departure from the normal practice, the pope will not return to Rome aboard a plane of the country he has visited, normally that country's nationalised flag carrier.

At a briefing for correspondents last week, Father Lombardi said the decision to fly there and back with the Italian airline, Alitalia, had been taken for reasons of "simplicity".

He added: "In any case, British Airways is no longer state-owned". Alitalia was itself, however, privatised two years ago.

In 2006, BA was embroiled in a bitter row after taking disciplinary action against an employee who refused to cover up a necklace carrying a cross which she wore on the outside of her uniform.

The last-minute withdrawal of the German cardinal has led to a flurry of activity in the Vatican, because he was to have played a central role in the ecumenical aspects of the pope's visit. Until July, Kasper was the head of the department that deals with relations with other Christian denominations, where he had worked since 1999.

His successor, a Swiss archbishop, Kurt Koch, speaks English, but he is not as fluent as Kasper. He also has far less experience of dealing with what Cardinal Kasper in his Focus interview called the "difficult dialogue" with the Church of England.

This is not the first time he has experienced controversy. Last year, after the pope lifted the excommunication of an ultra-traditional British bishop who had questioned the extent of the Holocaust, Kasper raised eyebrows in the Vatican with an outspoken interview in which he criticised a lack of consultation and said there had been "misunderstandings and management errors" in the papal bureaucracy.

Meanwhile in Britain, survivors of abuse within the Roman Catholic church repeated their demand that the pope hand over all information on abusive clergy.

They told a press conference in central London today that repeated apologies meant nothing and they wanted "truth, justice and accountability".

Peter Saunders, a survivor of Catholic abuse who is now the chief executive of the National Association for People Abused in Childhood (Napac), said: "We need the pope to say, 'I will hand over all the information I have about abusing priests wherever they are in the world. I will hand it over to the authorities of the countries where these people are being protected.'"

Earlier, more than 50 public figures signed a letter to the Guardian arguing that the pope should not be given the "honour" of a UK state visit.

The signatories, who included Stephen Fry, Terry Pratchett, Philip Pullman and Richard Dawkins call for "Pope Ratzinger" to be stripped of the right because of the Vatican's record on gay rights, abortion and birth control.