Survivors from an asylum seeker boat that sank off Indonesia this week are pleading to be taken to Australia.

The search for more survivors has been called off after the 55 people found alive were sent back to Indonesia, leaving nearly 100 people missing.

The boat, which sank in the Sunda Strait off the island of Java, was the fourth known to have sunk on that route to Christmas Island in recent years.

The survivors, who say they were fleeing persecution in Afghanistan, paid about $5,000 each to make the doomed journey.

In the West Java port of Merak, there were scenes of mayhem when Indonesian authorities brought the survivors back to the place they were trying to leave.

Indonesia is not a party to the UN Refugee Convention and does not offer protection to asylum seekers. Instead they are treated as illegal immigrants and often deported.

The asylum seekers claim they were told that they would be taken to Darwin for medical care until an Indonesian boat arrived.

"It was the Australian Navy officers who told us to get off their boat and get on this boat," one survivor, Muhammad Reza, told the ABC.

"First we thought it was sort of medical aid or something. But as we approached this boat, we could see the sign and then we started requesting them not to send us on this boat... please keep us on the Australian boat and take us there."

Another survivor, Muhammed Zahir, said he did not want to go back to Afghanistan.

"We came from Afghanistan, we don't like to go back [to] Afghanistan... We need to somebody to help us," he said.

"We lost 110 people, they're gone. They were from our family, my brother, my sister, they all gone. So how we can go back to Indonesia?"

Aside from that, the survivors say they were denied medical treatment and food on HMAS Maitland.

"I haven't slept for seven days already and I haven't eaten for seven days already," Mr Zahir said.

"I haven't [had a] drink [since] yesterday when I arrived on the Australian boat - I got water from a rain can and that's all."

'Tough situation'

Mr Zahir told the ABC the Australian Navy did not have enough food to provide it to the asylum seekers.

"They didn't have enough for everybody. They had for their own selves," he said.

And he said the survivors of the boat tragedy were not provided with medical treatment either.

"They said, 'we are not allowed to because we don't have any medical system or doctor or anything'," he said.

"Three days we were in the boat. After three days the boat broke... after four days of waiting, we were in the water, without boat, without lifejacket, without everything.

"We are in a tough situation. We didn't save medical things, we are all injured."

Indonesia's Human Rights Working Group, a non-governmental organisation, says Australia is neglecting its responsibilities to protect people under the Refugee Convention.

The group's head, Rafendi Djamin, is particularly critical of the planned indefinite detention on Nauru and Manus Island.

"It's a way of trying to find a loopholes in the existing international humanitarian law that gives protection, which is basically running away from the state obligation to protect them," he said.

He says asylum seekers will now have an avenue to file complaints against Australia for breaching human rights.

"They will have to be accountable on what they did. There is a process of... complaint which is available," he said.

"The Australian Government should not be surprised if there is a case filed in the international UN monitoring bodies on human rights," he said.

His organisation is among a number of international groups trying to get ASEAN countries, like Indonesia, to start offering temporary protection for refugees.

The ABC has been in touch with the Department of Defence this morning but so far there has been no official response to the allegations made by asylum seekers rescued by HMAS Maitland.

'No Darwin plans'

The Federal Government has rejected claims the asylum seekers were told they would be brought to Australia.

The survivors said the Australian Navy told them they would be taken to Darwin but Government frontbencher Brendan O'Connor says all protocols were followed.

"These matters are determined by masters of vessels and the agencies involved, the Indonesian agencies that were leading this rescue," he said.

"Of course, it was in the search and rescue zone of Indonesia, 42 nautical miles west of Java.

"So it's entirely appropriate and consistent with the protocols of search and rescue."

Opposition Immigration spokesman Scott Morrison says the asylum seekers should not be allowed to come to Australia.

"People rescued in the Indonesian search and rescue zone should be returned to Indonesia and that's what should be the outcome of the meetings of ministers in Indonesia next week," he said.