The housing secretary, James Brokenshire, has been angrily confronted by a Grenfell Tower survivor and accused of failing to do enough to prevent a repeat disaster.

The day after a fire raged through balconies at a housing block in Barking, Willie Thompson, who escaped the Grenfell disaster two years ago, warned Brokenshire that he would be held responsible if it happened again and said: “There’s another Grenfell in the post and it is going to land on your door.”

“You guys are doing almost nothing,” Thompson told the minister responsible for building safety at an event in parliament to mark the second anniversary of the fire that claimed 72 lives. “Does it take another Grenfell?”

“No,” Brokenshire said, who admitted the responsibility “weighs heavily on me” and later described the Barking fire as “deeply concerning”. “It takes a lot longer than we would have wanted it to. I accept your challenge on how we speed this up.”

“But I bet you are not losing any sleep at night,” Thompson, a father of two, angrily replied. “Parents are afraid their children will die … One of my best friends in the world was flashing his light [from his flat in Grenfell]. That man never came out. Take that image to bed with you.” Thompson then walked away.

The highly charged exchange came as survivors and the bereaved expressed frustration at the pace of changes to the culture over housing. On Sunday the Guardian revealed that a month before the fire at Barking Riverside, residents asked Bellway, the builder, to carry out fire safety checks, but were told there was nothing to worry about.

Ed Daffarn, who also escaped from Grenfell, told the reception attended by the home secretary, Sajid Javid, Conservative leadership hopeful Andrea Leadsom and Labour leader Jeremy Corbyn: “Just yesterday a fire ripped across flats in Barking. It is a miracle no one was harmed and it is distressing to us to hear residents were raising safety concerns before the fire, just like we were.

“In the days after the fire, the government promised that no stone would be left unturned in addressing the causes of the fire, ensuring that such a tragedy could never happen again. The survivors and the bereaved were promised justice and we were promised change. However, two years on, the reality is that little has changed and justice seems as far away as ever.”

Facebook Twitter Pinterest Firefighters examine the wreckage after the blaze at a block of flats in Barking, east London, on Sunday. Photograph: Dominic Lipinski/PA

Daffarn cited the amount of unsafe cladding still on hundreds of towers across the country, the failure to reform regulation of social housing, and repeated delays to the public inquiry, saying: “Justice delayed is justice denied.”

A group of survivors and bereaved are set to take legal action in US court against companies involved in making the cladding materials on Grenfell, including the US industrial giant Arconic, which made the rainscreen panels, and Celotex, which made the insulation. Celotex said it learned about the legal action on Friday and was “considering its position”. Arconic declined to comment.

Karim Mussilhy, whose uncle died in the fire, told MPs and ministers that survivors and the bereaved had to work “ridiculously hard just to get some of your attention, let alone have things change”.

“We shouldn’t be here,” he said. “We should be at home trying to rebuild the lives that were taken away from us, coming to terms with this new reality that we were forced into because we were ignored and neglected and lost our loved ones in the most horrific way possible. All we had is promises, promises and promises, broken promises.”

In a speech after being challenged by Thompson, Brokenshire said: “Two years have passed, and I know this has been testing in ways I cannot begin to imagine … I have to say that your strength and character and dignity has been truly humbling for me to see.

“For too long the people of Grenfell and people across the country have been excluded from conversations about the homes they are living in,” he said. “Whilst there has been progress on rehousing, I know there is a lot more to do. We need to continue to challenge ourselves.”

John Healey, Labour’s shadow housing secretary, said: “Not enough has been done to get every one of those Grenfell survivors into a new home, to get justice for the Grenfell community, to get all those other blocks with unsafe cladding removed and replaced, and also to get those far-reaching changes to building regulations.”