Pressure on Fiona Woolf to step down as the new head of the government inquiry into child abuse has intensified after a lawyer representing almost 50 victims said a number would not participate because of their concern about how it was going to be run.

The claims, by Alison Millar, a partner at Leigh Day solicitors, came as the leading organisation representing abuse survivors said many feared that it would be “a whitewash” and questioned Woolf’s credentials to lead the inquiry.

Woolf has been under scrutiny after it emerged that she had attended a number of social gatherings with Lord and Lady Brittan.

Lord Brittan’s role as home secretary in the 1980s is considered by abuse survivors to be crucial to the inquiry. They want to know what happened to a dossier about alleged Westminster paedophiles that went missing from his department. “As Leon Brittan is potentially such an important figure in this investigation, to appoint somebody who has demonstrably had more than a passing connection to the Brittans renders her unsuitable for the post,” said Peter Saunders, chief executive of the National Association for People Abused in Childhood.

“We’ve heard from many survivors who are extremely distressed. They are saying it’s going to be a whitewash, that it doesn’t want to speak to the victims and that’s just a paper exercise. They are worried it’s not going to get to the truth of what happened.”

Victims are dismayed that Woolf, who is to stand down as Lord Mayor of London next month, will be taking up another demanding job, separate from her role heading up the inquiry.

“Some of my clients question her capability in terms of her diary commitments to do this very complex and lengthy inquiry,” Millar said. “She’s due to take on a prestigious appointment with the University of Law, which will involve promoting it to international students. We want to know: does she have a lot of international trips booked, is she going to be able to come to the meetings? You cannot do this remotely.”

Saunders also questioned Woolf’s expertise. “She’s a corporate lawyer; she has no experience in investigating serious criminal cases,” he said.

Woolf is under pressure to release the draft of her letter sent to the Home Office outlining the social contacts she had with the Brittans. It is unclear whether the letter was then subject to revisions.

Keith Vaz, chair of the home affairs select committee, has asked Woolf to clarify a number of points regarding her appointment by next week.

One law firm has already drawn up plans to seek a judicial review of Woolf’s appointment. Millar suggested that that remained a possibility for her clients too. She said many had been upset by Woolf’s recent appearance before Vaz’s committee.

“One of the things that angered our clients was when Fiona Woolf started talking about the victim community,” Millar said. “There is no victim community; they are survivors of child abuse. They do not live in a communal state. They are a disparate range of people whose interests are not homogenous. Quite often they are isolated from other people, including each other, and they’re not all talking and saying ‘we want this we want that’. However, all the clients I’ve spoken with are unanimous that they don’t want Fiona Woolf to chair this inquiry because the perception of her is someone too close to the establishment.

“A number of my more vocal clients have said they’re not going to participate in the inquiry as it’s currently set up,” Millar said. “Whether they feel as strongly when the inquiry gets going I don’t know. But you’ve got a pretty sceptical group of people. If you set an inquiry up from day one that looks like it’s not going to listen to them, they are going to turn their back on it.”