A support group for victims of child sexual abuse is calling for one of the six people on the royal commission to resign or be removed from the panel.

International Survivors Network of Those Abused by Priests, also known as SNAP, is concerned that commissioner Robert Fitzgerald may have a conflict of interest.

Victims are worried they will not be able to speak openly about their experiences in front of Mr Fitzgerald because of his 30-year association with the Catholic Church.

SNAP's Nicky Davis says he is too close to the church to be sitting on the royal commission.

"He sits in some very prominent positions in Catholic organisations," she said.

"I think that this is a serious enough issue and is going to affect so many victims that it would be best if he resigned from the royal commission and was replaced with someone that does not have close ties to any organisation that's been involved in large scale child sexual abuse.

"Catholic victims don't trust anyone associated, closely associated with the Catholic Church."

Dr Fitzgerald says he no longer has any associations with the Church besides being a Catholic.

He says once he was appointed to the royal commission he resigned from all of his positions associated with the Catholic Church.

Federal Attorney General Mark Dreyfus has dismissed the call for his resignation.

"Robert Fitzgerald was selected following very broad consultation, a rigorous selection process, and the Government has got every confidence that he will perform his duties with integrity and with professionalism," he said.