(CNN) A confrontation is looming in Papua New Guinea (PNG) between local authorities and more than 700 men who are refusing to leave an Australian-run immigration processing center.

The center on Manus Island was set to be cleared Monday, ahead of its formal closure Tuesday, but many of the refugees and asylum seekers claim they'll be attacked if they leave the security of the compound's wire fences by locals who don't want them there.

Those at the center include 551 refugees and 167 other asylum seekers whose claims were rejected by the Australian government.

PNG Police Commissioner Gari Baki, in a statement, said, "The safety of both the refugees and government workers plus staff of leading agencies is not to be taken for granted given the tension that is now being expressed by the locals on Manus Island." He appealed to locals to let the transfer be done "as smoothly as possible."

Behrouz Boochani, a Kurdish refugee who has been in the facility since fleeing Iran in 2013, said the refugees are "extremely worried and scared" and would resist attempts to move them.

"The refugees don't feel safe in the community, because the local community is not ready to accept them," Boochani told CNN. "The refugees don't want to leave."

JUST WATCHED Behrouz Boochani and Arash Kamali Sarvestani co-directed a film for Sarvin Productions about Manus Island called "Chauka, Please Tell Us The Time." Replay More Videos ... MUST WATCH Behrouz Boochani and Arash Kamali Sarvestani co-directed a film for Sarvin Productions about Manus Island called "Chauka, Please Tell Us The Time." 01:46

Suspicion on both sides

The center sits on the Lombrum Naval Base, a 30-minute bus ride from the center of Lorengau. In April 2016, the PNG Supreme Court ruled the refugees, who were not allowed to leave the center, were being deprived of their personal liberty. Soon after, the once heavily-guarded gates were opened, allowing refugees to come and go.

The sudden presence of refugees, all men, from places like Iran, Sri Lanka, Afghanistan and Myanmar, in a town of just 6,000 people, caused suspicion and conflict, according to former Manus Island MP Ron Knight.

"It boils down to if everyone behaves themselves it's all good," he told CNN. "You have young men who have been locked up for four or five years. They get a taste of freedom. You have young women in town who are attracted to these guys. They keep these relationships hidden. The fathers find out, or the families find out, and it becomes a big issue."

The refugees and asylum seekers say they've been assaulted when they've ventured outside the center's gates. A Human Rights Watch report released Wednesday says "groups of local young men, often intoxicated and sometimes armed with sticks, rocks, knives, or screwdrivers, have frequently assaulted and robbed refugees and asylum seekers on Manus Island."

The Manus regional processing center is located on the Lombrum Naval Base, outside the main town of Lorengau.

Tim McKenna, a refugee advocate from St Vincent de Paul Society who traveled to Manus in early October to help support the men, said they were easy targets.

"These guys have mobile phones, quite good ones, because they need to speak to their families, so they're a target for that. At some stage people have been slashed, they've had knives stuck in their faces and so on," McKenna said.

Knight said Manus was "very quiet," before the detainees arrived. He blamed the trouble on PNG mainlanders who had moved to the island to work on construction projects for the former detainees.

"It's not a cultural trait of the Manus people to hold up people in the road and slash them with machetes. That has never happened in Manus in the past, never," Knight said.

Fears 'stoked by outsiders'

Lawyers acting for the refugees are planning to file an injunction in the PNG Supreme Court on Tuesday to stop the center's closure, "not because we want the center to stay open," said Melbourne-based barrister Greg Barns, "but because there is an absence of alternatives for these men."

The center was originally opened in 2001 as part of then Prime Minister John Howard's "Pacific Solution," to process asylum seekers in offshore facilities.

It was closed in 2008 by the Labor government, but reopened in 2012 after a rise in the number of boat arrivals to a peak of 300 in 2013, carrying more than 20,500 people . Two years later, the government announced that boat arrivals had stopped.

For months, the Australian government has been telling the men they must move out of the Manus facility, and services have been gradually cut to persuade them to leave. A small number have already left.

Water and power has already been cut to some compounds within the Manus Island detention center.

Torture and trauma counseling was canceled, the gym shut, the landline phone removed, and the canteen is due to close on Sunday.

When accused of attempting to "starve the men out," Immigration and Border Secretary Michael Pezzullo told a Senate Estimates hearing on Monday there was "perfectly reasonable, appropriate, accommodation and other supports for the necessities of life... available in alternative locations."

The men remaining in the center have been asked to move to two other locations -- the East Lorengau Refugee Transit Center or West Lorengau House -- both provided by the Australian government at a cost of up to $190 million (A$250 million dollars) a year. Asylum seekers who haven't been found to be refugees are being asked to move to a third location, Hillside House.

At the same hearing, Pezzullo said the mens' fears about moving had been "stoked and fueled by troublesome outsiders."

Shots fired into detention center

The refugees and asylum seekers say they have good reason to be worried, not only about Lorengau residents but local authorities.

On Good Friday in April this year, a fistfight between refugees and locals at a nearby soccer pitch escalated into shots being fired at the center by PNG Defence Force personnel, according to refugees and the police.

"Those that were involved in the incident were drunk. Their conduct is unethical and unacceptable and it really tarnished the reputation of a disciplined organization," Manus Island Police Commander, Senior Inspector David Yapu, told Radio NZ.

According to human rights advocates, the fear of attack is never far from the minds of the men who, having fled war and oppressive regimes, have been locked up for years with little indication of when and where they'll be settled.

2016 UN refugee agency report about Australia's off-shore immigration centers said, "the prolonged, arbitrary and indefinite nature of immigration detention in conjunction with a profound hopelessness in the context of no durable settlement options (had)... rendered them vulnerable to alarming levels of mental illness."

Six people have died at the Manus Island center since it opened in 2012. The first, Kurdish refugee Reza Barati, was beaten to death in a riot in the compound in February, 2014. The most recent deaths were two suicides in the last two months.

"We don't have control over anything over here. The only thing we have control over is resistance," said Amir Taghinia, a refugee who fled religious persecution in Iran, and was sent to Manus Island in 2013. He spoke to CNN by phone from the PNG capital, Port Moresby, where he's been sent for medical treatment.

Taghinia paid a human trafficker to take him to the Australian territory of Christmas Island, just as the country was tightening its maritime borders in response to a rise in the asylum seekers arriving by boat.

Newly elected Prime Minister Tony Abbott had vowed to "stop the boats," and announced that no asylum seekers who arrived in country's waters would ever be allowed to settle in Australia.

Amir Taghinia

In April this year, Australian Immigration Minister Peter Dutton insisted the policy would not be relaxed.

"The advocates can bleat all they want, they can protest all they want. We have been very clear those people are not going to settle in our country because that would restart the people trade," he told CNN affiliate Sky News.

'This is not a safe place for us'

Every day at 2 p.m. for more than 90 days, the refugees and asylum seekers have been staging silent protests in the Manus compound; they squat on the ground, their arms crossed above their heads. Some hold signs urging the Australian government to let them move to Australia.

"This is not a safe place for us, this is not a place we can live life," Taghinia said.

A silent protest happens at 2 p.m. every day within the confines of the center.

This month, the UN Refugee Agency urged Australia to "address the imminent humanitarian crisis" presented by the center's closure. It said, "appropriate steps to avoid further tragedy and harm to vulnerable people have not been taken."

Pezzullo, the Australian Immigration and Border chief, told the Senate Estimates hearing that talk of a human rights disaster was a "rather colorful appreciation of the situation."

"I don't think it will entail a mass loss of life. I don't think it will entail the loss of life at all," he said.

Manus Island refugees hold signs during a daily protest calling for the Australian government to find another solution.

The Australian government is giving the refugees the option to settle in Papua New Guinea -- where crime rates are among the highest in the world -- return to the country from which they fled, move to Cambodia or transfer to Nauru, a tiny island in Micronesia where conditions are arguably worse.

Fifty-four refugees from Manus and Nauru have already left under a refugee swap with the United States. The original deal was for 1,250 refugees; it's not clear how many more the US will accept as they must undergo a strict vetting process.

New Zealand had offered to take 150 refugees each year from the center, but the Australian government declined the offer on the basis that it could encourage people smugglers to "get back into business."

Incoming New Zealand Prime Minister Jacinda Ardern has said the offer remains on the table , and it's one that refugees are begging the Australian government to take.

The locals don't understand either why the deal was refused.

"The anger in the local community is not so much towards the asylum seekers but towards the Australian administration for allowing this thing to happen, and drag out," said Knight, the former Manus Island MP.

"The Australian government should see that these people have been held there for a long time now. And there has got to be another way. What's the big deal? Let them go to New Zealand."

It's unclear what will happen when police and immigration officials move in on Monday. If the center does close, as planned, the men say it's only a matter of time before more of them die -- by their own hand or someone else's.

"I just want safety," said Taghinia. "I want to live my life. I don't want to die on this island."