An advocate for asbestos victims has accused the Coalition of not caring about them and using the alleged unsafe removal of asbestos from National Broadband Network (NBN) sites to score political points.

Reports of asbestos not being disposed of properly by workers building the NBN surfaced last week with Telstra launching an audit of the work being carried out in all of their pits across the country. There have also been several crisis meetings held involving representatives from the government-owned NBN Co, Telstra, the federal government, unions and ADFA.

The Asbestos Diseases Foundation of Australia president, Barry Robson, said he was left “disgusted and upset” by the way the Coalition had handled the issue and told Guardian Australia parts of the media had whipped up the story.

“It’s being politicised by the Coalition when what should have been happening was the parties coming together saying ‘yes we’ve got a massive problem’ and in particularly two young families of Penrith who have been exposed to a possible, I’ll say it again, a possible death sentence hanging over their heads,” he said.

“And here we go, the opposition getting up trying to score cheap political points.

“I was thoroughly disgusted and upset.”

Robson said he had been particularly put off by the opposition’s performances in question time and that it was looking for political opportunities in what was a very serious issue.

He said the story had been misunderstood by a lot of people. He said the point was not that asbestos had been found in the pits, as Australia used to be one of the biggest consumers of the product and it was found “everywhere”.

“The story,” he said, “is sub-contractors working on a Telstra pit, not an NBN pit, a Telstra pit, removed asbestos in an unsafe manner and by doing so possibly contaminated two residencies.

“That’s the story. As simple as that. And now all of this other bloody gasbagging about NBN and who did this and who knew what is irrelevant.

“These two families’ health is more important than all of that.”

Robson accused Abbott of not making an effort as minister for health or minister for industrial relations to contact any asbestos victims’ group.

“I would like to hear them say to us ‘we are with you on this, this is a scourge on our nation, on our population and we’re going to do something about it with you’,” he said.

A spokesman for Abbott said the Coalition took the issue of asbestos contamination very seriously.

When asked if Abbott had met with asbestos victims’ groups while health or industrial relations minister he directed Guardian Australia to comments the opposition leader made during a doorstop on Tuesday.

“One of my early acts as minister for workplace relations was to commence the process for banning the importation and use of chrysotile asbestos,” Abbott said.

“As health minister I committed $6m to establish a national research centre into asbestos-related diseases.

“So, the record is there for all to see and my plea to the government now is please, please, given what we know about asbestos, make sure that the roll-out of the National Broadband Network doesn't add to the asbestos hazards that the Australian people face.”