The federal government is standing by its controversial Centrelink debt recovery system despite widespread criticism.

Human Services Minister Alan Tudge insists the automated process is not flawed, saying more than $300 million worth of overpayments has been recouped so far as part of an overall claw back of $4 billion.

"We are not going to scrap the welfare compliance system... (it's) working and we will continue with it," he told reporters in Canberra on Wednesday.

But Labor frontbencher Shayne Neumann argued it was "arrogant and offensive beyond belief" for the minister to come back from holidays and pretend there was no problem.

Mr Neumann said his office had helped one woman who was sent a debt letter claiming she owed $20,000 which was incorrect.

Mr Tudge acknowledged that in 20 per cent of cases people are able to explain the anomaly between self-reported income to Centrelink and tax office records.

But he disputed Labor's claim it was therefore a 20 per cent error rate.

"That is absolutely false, this is the system working as intended," Mr Tudge said.

Asked about Centrelink staff referring distressed people who had received debt notices to the counselling hotline Lifeline, Mr Tudge said it had always been the case that some people are referred to support, but it was completely disconnected to the debt recovery system.

The system has produced about 170,000 debt notices since July, based on automated data from the tax office and Centrelink.

But there are widespread complaints from distressed welfare recipients who claim they've been mistakenly targeted.

It's led to calls from Labor and the Greens to scrap the scheme, which is being investigated by the Commonwealth Ombudsman.

Mr Tudge, however, said he wasn't aware of anyone who was completely convinced they don't owe money but have been given a debt notice.

"There are three opportunities for individuals to update their records if a discrepancy is identified," he said.

He denied reports a man had to call Centrelink 350 times to try and resolve his matter, but acknowledged wait times can be long on occasions.

"Yes, sometimes people do have to wait longer than what they would like to wait. We simply ask them to be patient, that they will get through or they can go into a Centrelink office."

Mr Tudge reiterated that if someone has deliberately defrauded the welfare system, they would be tracked down and could face jail time.

The Australian Council of Social Service says the federal government has a duty of care towards people who call on it for support.

"We are hugely concerned that people are paying back debts that they do not owe because it is too hard to prove that they do not owe it," spokesman Peter Davidson said.