The second head of the government’s child abuse inquiry is facing calls to stand down over her social links to Lord Brittan, who was home secretary when a dossier about alleged Westminster paedophiles went missing from his department.

Fiona Woolf, a prominent solicitor and Lord Mayor of London, was brought in after Lady Butler-Sloss resigned from her role leading the inquiry when it emerged that her late brother Lord Havers was attorney general at the time of some of the historical allegations.

Ministers had hoped that Woolf would be able to draw a line under the previous controversy but a row erupted on Tuesday when it was revealed that Brittan was one of her neighbours, with whom she had dined five times since 2008. Woolf had also had coffee on “a small number of occasions” with Brittan’s wife (most recently last year), sat on a prize-giving panel with her and sponsored her for a fun run.

In a hearing with the House of Commons home affairs select committee, Woolf said there was “no close association” with the Tory peer and she believed her account of contact with him would “lay to rest” any fears. Stressing she was “not a member of the establishment”, she said: “I’ve gone the extra distance to make sure that I’ve dug out every possible connection … to make sure that it absolutely settles all the concerns that may be out there.”

However, Simon Danczuk, the Labour MP for Rochdale, who wrote a book about allegations of child abuse against the late Liberal MP Cyril Smith, said it was another example of the government picking an establishment figure for the job.

“She shouldn’t be chairing the inquiry,” he told the BBC’s World at One. “I’m very disappointed that they’ve chosen somebody who is so closely connected to Leon Brittan. He was the home secretary at a critical time when a lot of the abuse was going on ...

“I think she says that she’s been at dinner parties on a number of occasions with Leon Brittan. I don’t know what world she inhabits but where I come from if you’ve been to dinner at somebody’s home and vice versa then you’re relatively close. I would consider them friends. This suggests very little independence in terms of the chair.”

John Mann, Labour MP for Bassetlaw, also criticised the appointment, saying it was “totally impossible for Fiona Woolf to now properly chair child abuse inquiry” given that “Leon Brittan oversaw significant inquiries that vanished”.

The role of Brittan is key to the inquiry because it will investigate what happened to a dossier handed to him by the late MP Geoffrey Dickens, which later went missing. Last year the Tory peer said he could not remember getting the dossier, but after further questions were put to him in July, he released a statement saying he could now recollect a meeting with Dickens. He said he had asked officials to look into the claims and could not remembering hearing any more about it.

However, a Home Office review from last year found Brittan had written to Dickens in 1984 saying the material had been assessed by the director of public prosecutions as worth pursuing and “passed to the appropriate authorities”. Brittan released a second statement saying he had only just been made aware of last summer’s review, which proved that appropriate action had been taken.

Downing Street backed Woolf, saying it was confident she would ensure there was “no stone left unturned” as head of the inquiry. Theresa May, the home secretary, also gave her support, saying: “Fiona Woolf has a long and distinguished career throughout which she has demonstrated the highest standards of integrity.

“I am confident that she will lead the work of the panel with authority, and that under her leadership the panel will get to the truth of these issues.”

Woolf will be supported in her role by a panel of experts including Sharon Evans, Ivor Frank, Dame Moira Gibb, Professor Jenny Pearce, Dru Sharpling, Professor Terence Stephenson, Graham Wilmer, and Barbara Hearn, overseeing allegations from the 1970s to the present day. Ben Emmerson QC is counsel to the inquiry, and Professor Alexis Jay, who helped uncover the child abuse in Rotherham, will serve as an adviser.

In a sign of the difficulty in recruiting panel members with no “establishment links”, Gibb disclosed that she was a “personal friend” of Gerald Malone, a former minister who was a parliamentary private secretary to Brittan in 1985, and a government whip from 1986 to 1987.

• This article was amended on 24 October 2014. An earlier version erred in describing Fiona Woolf as a “prominent QC”; she is a solicitor.