Lady Butler-Sloss, the retired high court judge, has resigned as chair of the panel that is due to examine the extent to which public institutions failed to investigate allegations of child abuse after admitting that she had failed to take into account a family conflict of interest.

Hours after the former solicitor general Vera Baird called on Butler-Sloss to stand down because her brother served as attorney general in the 1980s, when reports of child abuse were allegedly not examined properly, the former judge issued a statement announcing that she would withdraw from the post.

Butler-Sloss said she had been honoured to be invited to chair the inquiry. But she added: "It has become apparent over the last few days, however, that there is a widespread perception, particularly among victim and survivor groups, that I am not the right person to chair the inquiry. It has also become clear to me that I did not sufficiently consider whether my background and the fact my brother had been attorney general would cause difficulties."

The retired judge had faced intense criticism from victims' groups because her brother, the late Sir Michael Havers, was attorney general during the 1980s – the period due be examined by the panel.

The retired judge informed the home secretary, Theresa May, of her decision over the weekend. May, who appointed Butler-Sloss last week, had strongly defended her as the criticism mounted.

Butler-Sloss added in her statement: "This is a victim-orientated inquiry and those who wish to be heard must have confidence that the members of the panel will pay proper regard to their concerns and give appropriate advice to government.

"Nor should media attention be allowed to be diverted from the extremely important issues at stake, namely whether enough has been done to protect children from sexual abuse and hold to account those who commit these appalling crimes.

"Having listened to the concerns of victim and survivor groups and the criticisms of MPs and the media, I have come to the conclusion that I should not chair this inquiry and have so informed the home secretary.

"I should like to add that I have dedicated my life to public service, to the pursuit of justice and to protecting the rights of children and families and I wish the inquiry success in its important work."

Theresa May said: "I am deeply saddened by Baroness Butler-Sloss's decision to withdraw but understand and respect her reasons. Baroness Butler-Sloss is a woman of the highest integrity and compassion and continues to have an enormous contribution to make to public life.

"As she has said herself, the work of this inquiry is more important than any individual and an announcement will be made on who will take over the chairmanship and membership of the panel as soon as possible so this important work can move forward."

Downing Street pointed out that it had been "entirely" the decision of Butler-Sloss to stand down. No 10 went out of its way to praise her work, describing her work as chair of the Cleveland child abuse inquiry as groundbreaking.

The prime minister's spokesman said: "Dame Elizabeth has taken the decision to step down as the head of the panel inquiry. It is entirely her decision. Understanding and respecting her decision, the government's view has not changed: that she would have done a first-class job as chair of the panel. But she has taken that decision."

Keith Vaz, the chairman of the Commons home affairs select committee who raised concerns about the appointment with the Home Office permanent secretary, Mark Sedwill, last week, said that the inquiry was now becoming shambolic.

Vaz said: "I am not surprised by this decision – it is the right one. As I pointed out to Mr Sedwill the public would be concerned that a member of parliament, no matter how distinguished, had been appointed to head this important panel. The whole inquiry process is becoming shambolic: missing files, ministers refusing to read reports and now the chair resigning before the inquiry is has even commenced."

Butler-Sloss's decision to stand down is a blow to the government, which appeared to have rushed into appointing her. On Sunday last week Michael Gove said there would be no public inquiry. Within 24 hours the home secretary announced a wide-ranging inquiry that will examine how public institutions responded to allegations of child abuse.

There were suggestions that the Home Office overlooked Butler-Sloss's family links. Government sources insisted last week that it was well known that Butler-Sloss was the sister of Havers, attorney general from 1979 to 1987.

Earlier on Monday Baird, the police and crime commissioner for Northumbria who served as Labour's solicitor general from 2007 to 2010, said May had made an error in appointing Butler-Sloss because of her brother's role. The Butler-Sloss panel would have had to examine whether Havers played down allegations of child abuse during that period.

Baird told the Today programme on BBC Radio 4: "If she were in a court case presiding over it and her brother were mentioned as someone she may have to investigate, she would of course withdraw due to a conflict of interest. The conflict of interest is even bigger here where we have a vulnerable community of people who say that they have been not allowed to get justice.

"It is her task to look into it. It has got to be done by somebody who is an outsider to this, who is completely independent. Without wanting to descend totally to cliche, justice must not only be done but it has to be seen to be done."