Last month, after more than four years of suffering, 25 excited and nervous male refugees left Papua New Guinea's remote Manus Island for a new life in the United States.

Left behind are another 800 men - refugees and asylum-seekers - who are wondering if they too will be rescued by the US government or remain in a dead-end limbo while exposed to violence in Papua New Guinea.

"We didn't know if the US deal was real or fake," a refugee from Pakistan told me. He has completed his US interview but is not among the 25 who left for the US. "Now we know, but I am still so worried. How many will the US take and when? And who will be left behind?"

For more than four years, Australia has warehoused these men in Papua New Guinea in harsh and dangerous conditions. Papua New Guinea has recognised most of them as refugees who fled persecution in their home countries. They attempted perilous boat journeys to Australia. Instead of finding refuge there, they have faced violent attacks by Papua New Guinean locals and security forces, inadequate medical care, and uncertainty about their future.

Australia's policy of offshore processing is designed to deter future boat migrants. But Manus Island has not turned out to be a welcoming or safe place for the refugees. Quite the opposite.

I visited Manus Island last week and met men who had been beaten, robbed, and stabbed by young men from the local Papuan community. I met refugee men so paralysed by fear that they rarely leave the guarded centres where they live.

One of them, 35-year-old Jahim (not his real name) from Bangladesh, told me what happened one June morning when he and a friend were in Lorengau town to buy a chicken. A group of local men cornered them, demanding they hand over their phones and belongings. Initially, Jahim resisted. A man put a knife to his throat. "I was too late in handing over the phone. It's the only way to contact my family. Then he cut me here." He shows me a long scar at his elbow.

READ MORE: First refugees held on Manus Island leave for US

Following the attack, Jahim went to the local hospital. He says hospital staff claimed they could not treat him, and only offered him some pain relief. So Jahim took a taxi to the main detention centre on a naval base, which maintains a medical clinic. From there, authorities sent Jahim by plane to Port Moresby hospital for surgery.

Nearly everyone I spoke to had witnessed or experienced violence or theft by local men, usually young, often intoxicated. The night before I arrived, a large group of local youths beat two Afghans near the transit centre. "I hadn't left the transit centre in three months because I don't feel safe," one of the refugees told me. "I went out and they wanted to fight us. They beat me with sticks, with rocks until I fell down dizzy. They took my phone, my money." His face was still swollen from the beating.

Papua New Guinea is not a safe place for any foreigner, but it is especially dangerous for young men without money or connections, many of whom do not speak English and have little experience of living in hostile foreign environments.

Refugees say Manus police have not held anyone to account for these robberies. Refugees have stopped reporting the cases to the police because of their inaction. Jahim told me, "I didn't tell the police. The police are not listening to us. It's not my country. Now when I see Papuans, I feel scared. I don't come to town."

The Australian government forcibly transferred these men to Manus Island. Now it is forcing them to leave the main detention centre by October 31, and has already started dismantling some buildings. Australian and Papua New Guinean officials say the men must move to a less-secure transit facility closer to town or else return to their home countries.

Supposedly, this is to implement a Supreme Court ruling from April 2015, which ruled detention of the men at the centre unconstitutional, forcing both governments to unlock the gates and provide some freedom of movement. I don't think another forcible transfer is what the judges had in mind.

It is a tragic irony that the only place on Manus Island these men feel safe right now is inside a centre where occasionally soldiers and police have shot at them, where locals once attacked them with machetes and beat them. But that centre, for now, is guarded and provides some security.

Papua New Guinea is not a safe place for any foreigner, but it is especially dangerous for young men without money or connections, many of whom do not speak English and have little experience of living in hostile foreign environments.

However grudgingly, the US government has done Australia a favour by agreeing to resettle some of the men from Manus Island and some of the 1,100 men, women and children from Nauru. But that's no excuse for Australia's inaction about those who remain.

Australia should immediately end offshore processing and resettlement by bringing every one of these men, women and children here to either have their asylum claims processed or if they are refugees, to restart their lives here. As Australia's partner, the US can help, by insisting that Australia should leave nobody behind in Papua New Guinea or Nauru. It can press Australia to accept offers to take some of the refugees that have been made by other generous third countries like New Zealand. Every day these men remain on Manus is another day of suffering.

Elaine Pearson is Australia director at Human Rights Watch. Follow her on twitter @pearsonelaine

The views expressed in this article are the author's own and do not necessarily reflect Al Jazeera's editorial policy.