Grenfell survivors have accused Boris Johnson of downgrading the government’s interest in the disaster and said he is out of touch with what they are going through.

As pressure rose on the prime minister to rescind his appointment of a public inquiry panel member revealed to have links to the combustible cladding company involved in the tower’s refurbishment, community leaders said: “Grenfell families have dropped down the agenda for Downing Street.”

Since the election last month there has been no government minister with specific responsibility for Grenfell victims, and the main bereaved and survivors’ group, Grenfell United, has yet to meet Johnson, having had three meetings with Theresa May during her premiership.

Concerns grew last week when Johnson was challenged by some community members to reverse his selection of Benita Mehra to help lead the public inquiry. The prime minister did not know that Mehra led an engineering industry charity that received a £71,000 grant from the charitable arm of Arconic, the firm that made the tower’s combustible cladding.

Mehra is due to help preside over inquiry hearings restarting on 27 January with an examination of the selection, testing and marketing of the panels found to have been the main cause of the fire’s spread.

It also emerged that an aide had to explain to Johnson that a tenant management organisation (TMO) was the tower’s landlord. The TMO ran the refurbishment project and is under investigation by Scotland Yard detectives exploring possible corporate manslaughter prosecutions.

On Friday, lawyers for some of the bereaved and survivors wrote to Johnson warning that if Mehra remained as one of two experts assisting the inquiry’s chairman, Sir Martin Moore-Bick, then public confidence in the inquiry process would be substantially damaged.

They said if he did not remove her then files showing the due diligence process that led to her selection must be released. The charity that Mehra led, the Women’s Engineering Society, has already supplied the government with details of the Arconic Foundation grant and Mehra’s role in helping to secure it.

Labour, Hillsborough families and the justice charity Inquest have backed survivors’ calls for Mehra’s removal from the panel. The inquiry has so far said it does not consider that her previous role “in any way affects her impartiality as a panel member”. A Cabinet Office spokesperson said there were “robust processes … [to ensure] any potential conflicts of interest are properly considered and managed”.

A Grenfell United spokesperson said: “It was apparent in the meeting last week that the prime minister is out of touch with what we’ve gone through the last few years. If we are ever going to see justice and real change for what happened to our 72 loved ones, it will be on Johnson’s watch. The shamefully poor due diligence shown by the government on the panel appointment has the potential of putting the entire inquiry process in question.”

Mohamed Ragab, whose nephew Hesham Rahman died in the fire, said he told Johnson in a meeting last Thursday that he used to work for the TMO.

“[Johnson] said: ‘What is the TMO?’ and one of the officials sitting next to him said it was the tenants’ management organisation,” said Ragab. “He hasn’t got any idea about the meaning of TMO.”

In October when Moore-Bick issued his findings from the first phase of the inquiry, Johnson told the House of Commons that the survivors and bereaved “have been let down many times before. Too often overlooked and ignored in the months and years before the tragedy.”

He said it was “the cladding that turned a routine and containable kitchen fire into a disaster of unprecedented proportions,” and it had been fitted in breach of building regulations.

“Why this was allowed to happen – and who was responsible for it – will be covered in phase two of his inquiry,” he said.

A Downing Street spokesperson said: “The prime minister is completely committed to doing whatever it takes to secure justice for the bereaved and survivors and all those affected by the tragic fire at Grenfell Tower. This government will ensure that they have an active and engaging role to play in implementing the lessons of Grenfell.”