Former high court judge criticised over fact her brother Lord Havers was attorney-general at time of some of controversy

Opposition to the appointment of Lady Butler-Sloss as chair of the panel of inquiry into child abuse is growing at Westminster with lawyers and key MPs, including the new chair of the health select committee, urging her to stand aside.

She has been criticised on the basis that her brother, the late Lord Havers, was attorney general at the time some of the controversy over the failure to prosecute child abuse cases may have occurred.

The Home Office insisted on Tuesday it would not ask her to step aside, but the possibility remains she will make her own decision to quit even before starting.

The former high court judge insisted she had been unaware of reports that her brother tried to prevent ex-MP Geoffrey Dickens airing claims about a diplomat in parliament in the 1980s.

"I know absolutely nothing about it," she told the BBC. "If people think I am not suitable, then that's up to them."

There is huge resentment among her friends that her suitability is being challenged.

But Sarah Wollaston, the chair of the health select committee, tweeted: "Not doubting her integrity but hard to see why Baroness Butler-Sloss would want to accept a role so many regard as conflicted at the outset."

Alison Millar, the lawyer who has represented many victims of child abuse, also voiced her concerns to the BBC, saying Butler-Sloss had enormous integrity but there needed to be not a shred of doubt that the inquiry was an establishment cover-up.

She said: "Survivors of abuse have a deep and well-founded distrust of authority figures given the breach of trust they have experienced as children. To enable them to be able to believe in the findings of this inquiry and for it to be seen, in their eyes, as a safe environment in which to talk about their experiences, there can be no shadows of doubt cast by links to allegations of an establishment cover-up.

"Whilst Baroness Butler-Sloss is a person of enormous integrity, the concern is that she is just too close to the establishment and in particular, concern over her family connection to Sir Michael Havers.

Labour MP Simon Danczuk who has played a leading role in calling for an investigation into child abuse allegations, also aired his doubts on Wednesday, saying it beggared belief that the government did not forsee the potential conflict of interest when it first invited her to take the post on Monday.

The decision to set up the inquiry was made over the weekend, and at the time Theresa May announced the inquiry on Monday afternoon no appointment of a chair had been made.

Butler-Sloss's experience in the family court made her appear at first the logical choice although her age raised questions about her stamina.

Danczuk acknowledged her expertise, saying she "does have a lot going for her in that she's very knowledgeable, very experienced, but there are these outstanding issues that raise concerns and we want somebody in the chair that exudes confidence and that's not the case".

He added: "She's part of the establishment and that raises concerns and the relationship in terms of her brother is too close for comfort. I think that's the conclusion most people will reach …

"The government should really look at this very closely. We want to get off to a flying start, we're not going to do that; we want to have confidence in the inquiry and this is not going to help matters. So I think the government should think again in terms of who they've appointed to this position."

The Labour MP said Lady Butler-Sloss should herself reconsider whether she was an appropriate appointment.

"I think she should consider her position. I find it quite surprising that neither she nor the government realised that her relationship with her brother was connected with Geoffrey Dickens and all this palaver around it. It beggars belief that that hadn't been considered in the first place."

He said that if ministers were unwilling to rethink the appointment, they needed to ensure the independent panel was made up of a diverse mix of people.

He emphasised that he was not questioning Butler-Sloss's credibility, but warned that her status as a peer also risked compromising the image of independence.

"I understand that, but she also sits in the House of Lords. Some of the people that may be accused of this type of abuse may sit in the House of Lords alongside her; that doesn't look good to the wider public."

A Home Office spokesman said: "Baroness Butler-Sloss has had a long and distinguished career at the highest levels of this country's legal system. Her work leading the Cleveland child abuse inquiry and as president of the high court's family division make her the perfect person to lead this important piece of work.

"As the permanent secretary told the home affairs select committee yesterday, the integrity of Baroness Butler-Sloss is beyond reproach and we stand by her appointment unreservedly."